Commit graph

144 commits

Author SHA1 Message Date
53341c70fc Change the locale codeset representation
From the normalized one, to the one actually contained within glibc. Recent
versions of glibc also contain symlinks linking the normalized codeset to the
locales with the .UTF-8 ending, but older ones do not.

Maybe handling codeset normalisation for queries would be good, but the locale
values ending in .UTF-8 are more compatible and allow the code to be
simplified. For querying, maybe there should be a locales table which handles
different representations.
2020-09-26 11:45:57 +01:00
af2e12a9ef Add some new metrics about load new revision jobs 2020-09-20 19:13:23 +01:00
Danjela Lura
7984179f5e Guard against null package metadata ids
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-08-13 08:58:09 +01:00
Danjela Lura
5b3e5d9441 Update the package_metadata_tsvectors with new entries
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-08-12 08:53:49 +01:00
a6a4a84b20 Avoid problems with lint warnings for duplicate packages
Use the deduplicated list of packages when fetching lint warnings, to avoid
duplicates. This was first seen in the following Guix commit
843344273c6a587b8e6c84d8406500fd64d8908a.
2020-08-09 16:53:26 +01:00
Danjela Lura
8d9a4d105c Get the translated package synopsis and descriptions into the database
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-06-19 16:03:32 +01:00
Danjela Lura
f67bea719d Get the translated lint checker descriptions into the database
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-06-07 20:17:03 +01:00
4d4ee801c0 Guard against strange derivations for packages
Where the requested system doesn't match that of the returned derivation. This
seems to happen for packages like dev86, wine and go.
2020-04-24 21:37:35 +01:00
d09a1ca791 Actually load the (guix lint) module if it's available 2020-04-20 14:29:33 +01:00
780ce66272 Adapt to the different inferior error handling
In recent versions of Guix on the Guix Data Service side, I don't think these
are system-error's, but a different type. For compatibility, just catch any
error.
2020-04-20 08:18:59 +01:00
c4969eb33f Improve detecting if the (guix lint) module is available
Previously it would cause an error when it's not available, which is a bit
unnecessary.
2020-04-20 08:18:19 +01:00
1b197884d0 Print out the system tests as the derivations are being computed
This might help with debugging failures in the load jobs around this point.
2020-04-07 20:08:26 +01:00
cdcc6140aa Guard against errors loading system tests 2020-04-07 18:22:13 +01:00
7db6a8fa03 Guard against errors when inserting log data
PostgreSQL seems to be unhappy with some data, I guess because it gets a bit
jumbled. Rather than failing the job, or getting stuck not inserting logs, try
and capture the error, log the details, and then keep going.
2020-04-07 17:38:57 +01:00
994318782a Pass #:store to checkers that support it
The derivation checker currently opens a store connection on its own, but by
passing the already open connection in, it won't have to do that, and
hopefully this will make checking all the packages faster.
2020-03-24 21:36:13 +00:00
ca0d3ee754 Stop using package_versions_by_guix_revision_range
It's been replaced by the package_derivations_by_guix_revision_range table.
2020-03-24 20:44:57 +00:00
c44297b615 Generate and store system test derivations for all supported systems
Rather than just the native system. I'm not quite sure of the value here, as I
guess system tests should behave the same regardless of the way the software
is compiled, but this seems like it could be useful, and being explicit about
the system the derivation is for is good.
2020-03-20 08:28:55 +00:00
e0f920bb14 Fix marking jobs as failed when exceptions are raised
The switch away from catch broke this, I obviously still don't quite get how
with-exception-handler works. Therefore, use it twice as that seems to help.
2020-03-19 20:43:37 +00:00
185ba36853 Log the time spent in a locked section 2020-02-28 20:41:44 +00:00
f23223f16d Improve the locking around fetching channel instances
I think there's still the potential for a race condition when working with the
cached checkout of the repository, but this lock was far too wide. The
sensitive section is just the latest-channel-instances call, as it adds the
relevant files to the store, and that's what's returned.
2020-02-28 20:40:31 +00:00
a95c5ca5c9 Improve the error reporting within process-load-new-guix-revision-job
By including a backtrace.
2020-02-27 21:14:58 +00:00
b166177dbd Prevent inlining for a number of procedures
So that the load-new-guix-revision tests work with Guile 3.
2020-02-27 21:14:34 +00:00
71f483c816 Add a few imports in preparation for Guile 3 2020-02-27 21:12:16 +00:00
2f41fe79be Adapt some license related code to work without mock in the tests
With Guile 3, there's a potential for mock to work in even fewer
circumstances. So, adapt the code to enable writing the tests without mock.
2020-02-24 21:47:34 +00:00
ce10833459 Switch to using some shorter syntax for logging time taken
In the load-new-guix-revision module.
2020-02-24 21:31:04 +00:00
f57b5c4a9a Add missing ROLLBACK for transaction 2020-02-24 18:50:34 +00:00
cee9acaa87 Refactor the logging setup out of process-load-new-guix-revision-job
To simplify both procedures.
2020-02-24 18:46:53 +00:00
36254f98e3 Improve the job logging
Switch to using a sequence for the ids in the log parts table, and spawn a
thread to listen for output from the inferior processes, and enter it in to
the database.
2020-02-15 17:42:07 +00:00
40f6de27f6 Make deleting the existing log parts optional 2020-02-15 16:49:37 +00:00
46c8ce5b82 Define a new parameter for a port used for the inferior error output 2020-02-15 13:41:17 +00:00
f6f51bbe0d Fake the store connection in the tests
So that the tests don't require a store connection.
2020-02-14 09:18:28 +00:00
5410961f65 Add temporary roots for channel instance derivations 2020-02-13 23:11:36 +00:00
0ef3349ed8 Use a more long lived store connection for loading data
As this will enable registering temporary roots, to avoid store items being
garbage collected.
2020-02-13 23:11:36 +00:00
9be7dbac0b Start storing channel instance derivations
These are the ones that relate to Guix pull.
2020-02-11 08:56:24 +00:00
406aa5e160 Switch using set-current-system to parameterize
As this is neater.
2020-02-11 08:55:34 +00:00
e0ee3d224b Refactor how channel derivations are handled
Yet again...

This makes the channel-derivations for each system accessible within the
load-new-guix-revision procedure, in preparation for storing them in the
database.
2020-02-09 21:40:56 +00:00
60d53f898f Collpase store-item-for-channel in to channel->guix-store-item 2020-02-09 21:06:12 +00:00
13b0ebe561 Start computing channel instance derivations for multiple systems
These aren't stored yet, but this is a start.
2020-02-09 21:03:05 +00:00
beab4babac Further refactor channel->derivation-file-name
To reduce the line length.
2020-02-09 16:44:18 +00:00
9c38115c3d Refactor channel->derivation-file-name to reduce line length 2020-02-09 16:39:46 +00:00
3ee481110e Shift extracting channel news to later on
To avoid race conditions between jobs, have it happen after the call to
extract-information-from.
2020-02-09 10:20:36 +00:00
f1989e8758 Rework cross derivations support
Stop using the system values as targets, and remove package derivation entries
where this is the case.

Switch the non-cross derivation case to have a target of "", rather than
matching the system, as this makes more sense, and is more consistent now that
the target values no longer match the system values.

Hardcode some more correct target values, and use these instead. Hopefully
this can be better integrated with Guix in the future.

This commit also includes a migration attempting to shrink some indexes.
2020-02-08 11:16:08 +00:00
8f0d5b51b1 Store information about system tests 2020-02-03 09:57:02 +01:00
65850669a0 Log the time to lock package_versions_by_guix_revision_range 2020-02-02 11:14:35 +01:00
908f6d3eee Log the time to lock load-new-guix-revision-inserts 2020-02-02 11:14:35 +01:00
956aa9ecb0 Log the time taken to acquire advisory session locks 2020-02-02 10:10:25 +01:00
f59354ed23 Call add-temp-root for the derivation file names
In an attempt to stop the derivations from being garbage collected between the
time they're generated, and when they're potentially read by the Guix Data
Service.
2020-02-02 08:30:02 +01:00
ee82c2c71d Refactor processing lint warnings
Extract out a function to make the code slightly easier to read.
2020-02-01 10:32:49 +01:00
1ab6ecd64f Use the line numbers to make the package deduplication more stable
Previously this would just compare on the version if the name was the same,
but there are package definitions that share the name and version (itstool is
one example).

To try and make this more stable, to avoid weird errors, and unstable
comparisons between revisions, use the line number when deduplicating
packages.
2020-01-21 23:55:36 +00:00
a66cbd41f9 Fail early if a package lookup fails when loading a revision
I think this might be happening when packages are filtered out as
duplicates (by name and version), but then a reference to a duplicate occurs
somewhere, like in a lint warning.
2020-01-21 23:53:52 +00:00