bdcs

Tools for managing a content store of software packages https://github.com/weldr/bdcs

Latest on Hackage:0.6.1

This package is not currently in any snapshots. If you're interested in using it, we recommend adding it to Stackage Nightly. Doing so will make builds more reliable, and allow stackage.org to host generated Haddocks.

LGPL-2.1 licensed by Chris Lumens
Maintained by clumens@redhat.com

Build Status Coverage Status

This code generates a metadata database (mddb) given an input directory of RPMs. You can generate this either by running locally or running under docker. It’s really best if you have the RPMs stored locally, too, not under some NFS mount or other network storage. That can slow things down quite a bit.

Importing the same set of RPMs into the same database twice should result in no changes. Importing additional RPMs into the same database should result in those RPMs being added to the existing database. There is currently no provision for removing an imported RPM. In this way, you could import a very large set of packages piecemeal if needed.

Running locally

You will first need a directory full of RPMs somewhere. Here, I assume that is the $PWD/Packages directory. Then run:

$ cabal sandbox init
$ cabal install --dependencies-only --enable-tests
$ cabal build
$ sqlite3 metadata.db < schema.sql
$ for f in ${PWD}/Packages/*rpm; do dist/build/bdcs-import/bdcs-import metadata.db cs.repo file://${f}; done

Running with docker

Running with docker is a two step process, as indicated by Dockerfile.build and Dockerfile. Dockerfile.build is used to compile the program needed to build an mddb and produces an image with that program. Dockerfile then runs that image and produces the mddb.

The Dockerfile depends on a base image, named welder/fedora:latest, which needs have been previously built. If it is not available it can be built from the welder-deployment repository by running make weld-fedora.

The Makefile lays out the exact steps and can be used to simplify all this - just run make importer mddb. If make is unavailable, just copy the steps out of there and run them manually.

The Makefile expects that the RPMs are in $PWD/rpms.

After completion, the mddb and content store will be in a bdcs-mddb-volume docker volume.

Preparing local development environment for Haskell

For development we use the latest upstream versions:

  1. Remove the standard haskell-platform and ghc-* RPMs if you have them installed
  2. Download version 8.0.2 of the generic Haskell Platform distribution from https://www.haskell.org/platform/linux.html#linux-generic
$ tar -xzvf haskell-platform-8.0.2-unknown-posix--minimal-x86_64.tar.gz
$ sudo ./install-haskell-platform.sh
  1. Add /usr/local/bin to your PATH if not already there!
  2. Install build dependencies:
# dnf -y install xz-devel zlib-devel glib2-devel gobject-introspection-devel ostree-devel

NOTE: On RHEL 7 ostree-devel is part of the Atomic Host product!

Building the project locally

cabal is used to install and manage Haskell dependencies from upstream.

$ cd src/ && cabal sandbox init && cabal install

Executing unit tests

$ cabal sandbox init
$ cabal install --dependencies-only --enable-tests
$ cabal test
Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to: dist/test/db-0.1.0.0-test-db.log
1 of 1 test suites (1 of 1 test cases) passed.

Produce code coverage report

$ cabal sandbox init
$ cabal install --enable-tests --enable-coverage
$ cabal test
$ firefox ./dist/hpc/vanilla/tix/*/hpc_index.html

Changes

0.6.1

  • Only support building with cabal-2.x. This is needed for private scope support on executables, which is needed for packaging.
  • Update URLs to RPMs used in testing.

0.6.0

  • Use the typed-process module instead of process.
  • Build all programs with -threaded. All users of bdcs will need to do the same.
  • Remove script support from the database. This wasn’t being used anyway.
  • Make the upstream_vcs field in the projects table optional.

0.5.0

  • Add logging of exceptions in directorySink, runCustomizations, runHacks, and runTmpfiles. This requires adding MonadBaseControl constraints to all these functions and MonadError constrains to those functions that did not already have it.
  • Require a filesystem package in the list of things to export.
  • Require a dracut package in the list of things to export for ostree.
  • Don’t assume /etc/shadow exists in the output artifact.
  • Log the call to dracut in ostreeSink.

0.4.0

  • Add BDCS.Projects.getProjectsLike, which returns projects whose names match the % and _ SQL wildcards.
  • Add BDCS.Projects.getProjectsTotal, which returns the number of projects in the database.
  • Add the BDCS.Export.Types module, which is useful for specifying what form the export artifact will take.
  • BDCS.Export.export and BDCS.Export.exportAndCustomize now require an ExportType parameter.
  • Remove BDCS.Export.Utils.supportedOutputs. The supportedExportTypes and exportTypeText functions in BDCS.Export.Types can be used to produce the same result.
  • The bdcs command line tool’s export subcommand now requires a -t argument for specifying the form of the export artifact. The destination argument is now also given with -d, instead of just bare on the command line.

0.3.0

  • Add BDCS.Groups.getGroupsLike, which returns groups whose names match the % and _ SQL wildcards.
  • Add BDCS.Groups.getGroupsTotal, which returns the number of groups in the database.
  • Add BDCS.DB.firstListResult, which returns the first value from the non-empty result list of an SQL query.
  • The types of many export-related functions have been modified to include a constraint on MonadLoggerIO.
  • Debug logging added to exportAndCustomize, runCustomizations, ostreeSink, qcow2Sink, and runHacks.

0.2.4

  • Allow building with aeson-1.3 and unordered-containers-0.2.9.

0.2.3

  • Add BDCS.Builds.findBuilds.
  • Add BDCS.Sources.findSources.

0.2.2

  • Add BDCS.CS.fileToObjectC, which, when used as part of a conduit, fetches the data for a single file from a ContentStore.
  • Add BDCS.Export.Customize, which provides types and functions for modifying the data exported from a ContentStore.
  • Add BDCS.Export.exportAndCustomize, which includes Customization data in an export.
  • Include schema.sql in the data-files.
  • Bug fixes related to FSTree and symlinks.

0.2.1

  • Add BDCS.Export.export, which is the bulk of the “bdcs export” command but in function form.
  • Add BDCS.Exports.Utils.supportedOutputs, which returns a list of supported output formats.
  • Ignore Enhances, Suggests, Recommends, and Supplements weak requirements during dependency solving.
  • Many test-related updates.
  • Allow building with conduit-extra-1.2.
  • Many docker-related build updates.
  • Support building with cabal-2.0.

0.2.0

  • Add a module for building virtual filesystem trees.
  • Remove use of partial functions.
  • baseURI now returns a Maybe.
  • getDbVersion now throws an error if there’s no version in the database.
  • Add a BadName exception that can be thrown by mkProject.

0.1.1

  • Add a new projects function that returns a list of all projects.
  • Projects is now an instance of FromJSON and ToJSON.
  • Fix running cabal commands under Docker.
  • Move where the epoch appears in the output of groupIdToNevra.
  • Add depcloseGroupIds for dependency solving from a list of IDs.
  • Add depcloseNames for dependency solving from a list of package names.
  • Rename depclose to depcloseNEVRAs to make its function clearer.

0.1.0

  • Initial release.
comments powered byDisqus