pantry
Content addressable Haskell package management
https://github.com/commercialhaskell/pantry#readme
Version on this page: | 0.9.3.1@rev:2 |
LTS Haskell 22.36: | 0.9.3.2 |
Stackage Nightly 2024-10-06: | 0.10.0@rev:1 |
Latest on Hackage: | 0.10.0@rev:1 |
pantry-0.9.3.1@sha256:4a79a5a9f1763c32ccad784d3a39515fe128611d675b621a628fc3d0aa2a5683,7863
Module documentation for 0.9.3.1
- Pantry
- Pantry.Internal
- Pantry.SQLite
pantry
Content addressable Haskell package management, providing for secure, reproducible acquisition of Haskell package contents and metadata.
What is Pantry
- A Haskell library, storage specification, and network protocol
- Intended for content-addressable storage of Haskell packages
- Allows non-centralized package storage
- Primarily for use by Stackage and Stack, hopefully other tools as well
Goals
- Efficient, distributed package storage for Haskell
- Superset of existing storage mechanisms
- Security via content addressable storage
- Allow more Stackage-style snapshots to exist
- Allow authors to bypass Hackage for uploads
- Allow Stackage to create forks of packages on Hackage
TODO
Content below needs to be updated.
- Support for hpack in PackageLocationImmutable?
Package definition
Pantry defines the following concepts:
- Blob: a raw byte sequence, identified by its key (SHA256 of the contents)
- Tree entry: contents of a single file (identified by blob key)
and whether or not it is executable.
- NOTE: existing package formats like tarballs support more sophisticated options. We explicitly do not support those. If such functionality is needed, fallback to those mechanism is required.
- Tree: mapping from relative path to a tree entry. Some basic
sanity rules apply to the paths: no
.
or..
directory components, no newlines in filepaths, does not begin with/
, no\\
(we normalize to POSIX-style paths). A tree is identified by a tree key (SHA256 of the tree’s serialized format). - Package: a tree key for the package contents, package name,
version number, and cabal file blob key. Requirements: there must be
a single file with a
.cabal
file extension at the root of the tree, and it must match the cabal file blob key. The cabal file must be located atpkgname.cabal
. Each tree can be in at most one package, and therefore tree keys work as package keys too.
Note that with the above, a tree key is all the information necessary to uniquely identify a package. However, including additional information (package name, version, cabal key) in config files may be useful for optimizations or user friendliness. If such extra information is ever included, it must be validated to concur with the package contents itself.
Package location
Packages will optionally be sourced from some location:
- Hackage requires the package name, version number, and revision number. Each revision of a package will end up with a different tree key.
- Archive takes a URL pointing to a tarball (gzipped or not) or a ZIP file. An implicit assumption is that archives remain immutable over time. Use tree keys to verify this assumption. (Same applies to Hackage for that matter.)
- Repository takes a repo type (Git or Mercurial), URL, and commit. Assuming the veracity of the cryptographic hashes on the repos, this should guarantee a unique set of files.
In order to deal with megarepos (repos and archives containing more
than one package), there is also a subdirectory for the archive and
repository cases. An empty subdir ""
would be the case for a
standard repo/archive.
In order to meet the rules of a package listed above, the following logic is applied to all three types above:
- Find all of the files in the raw location, and represent as
Map FilePath TreeEntry
(or equivalent). - Remove a wrapper directory. If all filepaths in that
Map
are contained within the same directory, strip it from all of the paths. For example, if the paths arefoo/bar
andfoo/baz
, the paths will be reduced tobar
andbaz
. - After this wrapper is removed, then subdirectory logic is applied,
essentially applying
stripPrefix
to the filepaths. If the subdir isyesod-bin
and files exist calledyesod-core/yesod-core.cabal
andyesod-bin/yesod-bin.cabal
, the only file remaining after subdir stripping would beyesod-bin.cabal
. Note that trailing slashes must be handled appropriately, and that an empty subdir string results in this step being a noop.
The result of all of this is that, given one of the three package locations above, we can receive a tree key which will provide an installable package. That tree key will remain immutable.
How tooling refers to packages
We’ll get to the caching mechanism for Pantry below. However, the recommended approach for tooling is to support some kind of composite of the Pantry keys, parsed info, and raw package location. This allows for more efficient lookups when available, with a fallback when mirrors don’t have the needed information.
An example:
extra-deps:
- name: foobar
version: 1.2.3.4
pantry: deadbeef # tree key
cabal-file: 12345678 # blob key
archive: https://example.com/foobar-1.2.3.4.tar.gz
It is also recommended that tooling provide an easy way to generate such complete information from, e.g., just the URL of the tarball, and that upon reading information, hashes, package names, and version numbers are all checked for correctness.
Pantry caching
One simplistic option for Pantry would be that, every time a piece of data is needed, Pantry downloads the necessary tarball/Git repo/etc. However, this would in practice be highly wasteful, since downloading Git repos and archives just to get a single cabal file (for plan construction purposes) is overkill. Instead, here’s the basic idea for how caching works:
- All data for Pantry can be stored in a SQL database. Local tools like Stack will use an SQLite database. Servers will use PostgreSQL.
- We’ll define a network protocol (initially just HTTP, maybe extending to something more efficient if desired) for querying blobs and trees.
- When a blob or tree is needed, it is first checked for in the local SQLite cache. If it’s not available there, a request to the Pantry mirrors (configurable) will be made for the data. Since everything is content addressable, it is safe to use untrusted mirrors.
- If the data is not available in a mirror, and a location is provided, the location will be downloaded and cached locally.
We may also allow these Pantry mirrors to provide some kind of query interface to find out, e.g., the latest version of a package on Hackage. That’s still TBD.
Example: resolving a package location
To work through a full example, the following three stanzas are intended to have equivalent behavior:
- archive: https://example.com/foobar-1.2.3.4.tar.gz
- name: foobar
version: 1.2.3.4
pantry: deadbeef # tree key
cabal-file: 12345678 # blob key
archive: https://example.com/foobar-1.2.3.4.tar.gz
- pantry: deadbeef
The question is: how does the first one (presumably what a user would want to enter) be resolved into the second and third? Pantry would follow this set of steps:
- Download the tarball from the given URL
- Place each file in the tarball into its store as a blob, getting a blob key
for each. The tarball is now represented as
Map FilePath BlobKey
- Perform the root directory stripping step, removing a shared path
- Since there’s no subdirectory: no subdirectory stripping would be performed
- Serialize the
Map FilePath BlobKey
to a binary format and take its hash to get a tree key - Store the tree in the store referenced by its tree key. In our example: the
tree key is
deadbeef
. - Ensure that the tree is a valid package by checking for a single cabal file
at the root. In our example, that’s found in
foobar.cabal
with blob key12345678
. - Parse the cabal file and ensure that it is a valid cabal file, and that its
package name is
foobar
. Grab the version number (1.2.3.4). - We now know that tree key
deadbeef
is a valid package, and can refer to it by tree key exclusively. However, including the other information allows us to verify our assumptions, provide user-friendly readable data, and provide a fallback if the package isn’t in the Pantry cache.
More advanced content discovery
There are three more advanced cases to consider:
- Providing fall-back locations for content, such as out of concern for a single URL being removed in the future
- Closed corporate setups, where access to the general internet may either be impossible or undesirable
- Automatic discovery of missing content by hash
The following extensions are possible to address these cases:
- Instead of a single package location, provide a list of package locations with fallback semantics.
- Corporate environments will be encouraged to run a local Pantry mirror, and configure clients like Stack to speak to these mirrors instead of the default ones (or in addition to).
- Provide some kind of federation protocol for Pantry where servers can registry with each other and requests for content can be pinged to each other.
Providing override at the client level for Pantry mirror locations is a MUST. Making it easy to run in a corporate environment is a SHOULD. Providing the fallback package locations seems easy enough that we should include it initially, but falls under a SHOULD. The federated protocol should be added on-demand.
Changes
Changelog for pantry
v0.9.3.1
- Depend on
aeson-warning-parser-0.1.1
.
v0.9.3
- Add error S-628 (
LocalNoArchiveFileFound
). - Depend on
rio-prettyprint-0.1.7.0
.
v0.9.2
defaultCasaRepoPrefix
references https://casa.stackage.org, instead of https://casa.fpcomplete.com.- Depend on
crypton
instead ofcryptonite
. - Depend on
tar-conduit-0.4.0
, which will tolerate long filenames and directory names in archives created bygit archive
.
v0.9.1
- Expose module
Pantry.SQLite
.
v0.9.0
- Remove module
Pantry.Internal.AesonExtended
and depend onaeson-warning-parser
package. - Remove module
Pantry.Internal.Companion
and depend oncompanion
package. - Remove module
Pantry.Internal.StaticBytes
and depend onstatic-bytes
package. - Remove module
Pantry.Internal
, previously exposed only for testing. - Update Hackage bootstrap root key set.
v0.8.3
- Expose
withPantryConfig'
, which allows for optional use of Casa.NoCasaConfig
is now a data constructor ofPantryException
. withRepo
, in the case of Git, will now, if necessary, fetch the specific commit. (For example, GitHub repositories include the commits of unmerged pull requests but these are not fetched when the repository is cloned.)
v0.8.2.2
- Add error S-395 (
NoLocalPackageDirFound
).
v0.8.2.1
- On Windows, avoid fatal
tar: Cannot connect to C: resolve failed
bug when archiving repository submodules.
v0.8.2
PantryException
is now an instance of theText.PrettyPrint.Leijen.Extended.Pretty
class (provided by therio-prettyprint
package).- Module
Pantry
now exportsFuzzyResults
,Mismatch
andSafeFilePath
(andmkSafeFilePath
), used in data constructors ofPantryException
.
v0.8.1
- Support
hpack-0.35.1
, and prettierHpackLibraryException
error messages.
v0.8.0
findOrGenerateCabalFile
,loadCabalFilePath
,loadCabalFile
andloadCabalFileRaw
no longer assume that the program name used by Hpack (the library) is “stack”, and take a new initial argument of typeMaybe Text
to specify the desired program name. The default is “hpack”.
v0.7.1
- To support the Haskell Foundation’s
Haskell Error Index initiative, all Pantry
error messages generated by Pantry itself begin with an unique code in the
form
[S-nnn]
, wherennn
is a three-digit number.
v0.7.0
- Change
defaultHackageSecurityConfig
such that fieldhscIgnoreExpiry = True
, to be consistent with the defaults of theWithJSONWarnings HackageSecurityConfig
instance ofFromJSON
.
v0.6.0
- Rename
HackageSecurityConfig
asPackageIndexConfig
,defaultHackageSecurityConfig
asdefaultPackageIndexConfig
, andpcHackageSecurity
field ofPantryConfig
aspcPackageIndex
. - Expose new
HackageSecurityConfig
anddefaultHackageSecurityConfig
. The former represents Hackage Security configurations (only - no download prefix). - Change the data constructor of
PackageIndexConfig
to have fields for a download prefix (typeText
) and of typeHackageSecurityConfig
. - The
WithJSONWarnings PackageIndexConfig
instance ofFromJSON
now assigns default valuedefaultHackageSecurityConfig
if thehackage-security
key is absent from the JSON object. - Expose
defaultDownloadPrefix
, for the official Hackage server.
v0.5.7
- Expose
loadAndCompleteSnapshotRaw'
andloadAndCompleteSnapshot'
, which allow the toggling of the debug output of the raw snapshot layer. See #55. - Support GHC 9.4.
v0.5.6
- Remove operational and mirror keys from bootstrap key set. See #53.
v0.5.5
- Support
Cabal-3.6.0.0
.
v0.5.4
- Support
aeson-2.0.0.0
.
v0.5.3
- improve and expose
fetchRepos
/fetchReposRaw
.
v0.5.2.3
- Support for GHC 9.0. See #39.
v0.5.2.2
- Support for
Cabal-3.4.0.0
. See #38.
v0.5.2.1
- Support
persistent-2.13.0.0
. See #35.
v0.5.2
- Fall back to BSD tar when type cannot be detected. See #33.
v0.5.1.5
- Switch back to
hackage.haskell.org
. See #30. - Pass through basic auth credentials specified in URLs. See #32.
v0.5.1.4
- Allow building with
persistent-2.11.0.0
. See #28.
v0.5.1.3
- Handle case where tree exists in cache by blobs are missing. See #27.
v0.5.1.2
- Skip a test for issue #26.
v0.5.1.1
- Fix to allow multiple relative path of symlink.
v0.5.1.0
- Catch all exceptions from Casa calls and recover.
v0.5.0.0
- Make the location of LTS/Nightly snapshots configurable.
v0.4.0.1
- Removed errant log message.
v0.4.0.0
- Add a deprecation warning when using a repo/archive without a Cabal file, see Stack issue #5210.
- Do not include repo/archive dependencies which do not include Cabal files in lock files.
- Remove some no longer used functions.
v0.3.0.0
- Upgrade to
Cabal-3.0.0.0
.
v0.2.0.0
Bug fixes:
- Don’t compare the hashes of Cabal files. Addresses bugs such as Stack issue
#5045. Data type
changes: removed the
pmCabal
andrpmCabal
fields.
v0.1.1.2
Bug fixes:
- Module mapping insertions into the database are now atomic. Previously, if you SIGTERMed at the wrong time while running a script, you could end up with an inconsistent database state.
v0.1.1.1
Other changes:
- Support building with
persistent-template-2.7.0
.
v0.1.1.0
Changes since 0.1.0.0
Bug fixes:
- Fix to allow dependencies on specific versions of local git repositories. See Stack pull request #4862.
Behavior changes:
- By default, do not perform expiry checks in Hackage Security. See Stack issue #4928.
Other changes:
- Rename
pantry-tmp
package back topantry
, now that we have gained maintainership (which had been used by someone else for a candidate-only test that made it look like the name was free but prevented uploading a real package).
0.1.0.0
- Initial release.