Commit graph

154 commits

Author SHA1 Message Date
bd8390673e Fix squee upgrade issues in the load-new-guix-revision module 2021-01-02 11:13:30 +00:00
64a4058cce Start to add compatibility with squee returning #f for null values
While maintaining compatibility for older versions of squee.
2021-01-02 10:06:27 +00:00
276c4ecd71 Vacuum the log parts table after deleting entries from it
To help keep the number of dead tuples down.
2020-11-17 20:49:47 +00:00
e07b3291e8 Vacuum the package derivations table after each job
As each job deletes lots of rows in the table, vacuum it to free up space for
the next job.

I don't think this is something to do widely, but I wanted to try it out for
the tables which have a lot of churn.
2020-11-17 20:47:16 +00:00
6c47212c4d Improve the failed comparison page 2020-10-23 16:23:16 +01:00
4f3be14c16 Split the jobs metric in to succeede, queued and failed
Rather than just completed or not.
2020-10-09 20:19:28 +01:00
6023aa8caf Improve select-job-for-commit
It now returns #f or a list, rather than the empty list or a nested list.
2020-10-09 19:31:51 +01:00
da8586f62d Guard against errors when recording job stderr output
This might help at least handle errors when inserting data in to PostgreSQL.
2020-10-09 19:28:06 +01:00
03ea66cf33 Use the git repository fetch with authentication value 2020-10-07 18:51:30 +01:00
6bc1da014f Better handle loading the (guix i18n) module in the inferior
Previously it would only be loaded if the (guix lint) module exists.
2020-09-26 16:07:53 +01:00
53341c70fc Change the locale codeset representation
From the normalized one, to the one actually contained within glibc. Recent
versions of glibc also contain symlinks linking the normalized codeset to the
locales with the .UTF-8 ending, but older ones do not.

Maybe handling codeset normalisation for queries would be good, but the locale
values ending in .UTF-8 are more compatible and allow the code to be
simplified. For querying, maybe there should be a locales table which handles
different representations.
2020-09-26 11:45:57 +01:00
af2e12a9ef Add some new metrics about load new revision jobs 2020-09-20 19:13:23 +01:00
Danjela Lura
7984179f5e Guard against null package metadata ids
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-08-13 08:58:09 +01:00
Danjela Lura
5b3e5d9441 Update the package_metadata_tsvectors with new entries
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-08-12 08:53:49 +01:00
a6a4a84b20 Avoid problems with lint warnings for duplicate packages
Use the deduplicated list of packages when fetching lint warnings, to avoid
duplicates. This was first seen in the following Guix commit
843344273c6a587b8e6c84d8406500fd64d8908a.
2020-08-09 16:53:26 +01:00
Danjela Lura
8d9a4d105c Get the translated package synopsis and descriptions into the database
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-06-19 16:03:32 +01:00
Danjela Lura
f67bea719d Get the translated lint checker descriptions into the database
Signed-off-by: Christopher Baines <mail@cbaines.net>
2020-06-07 20:17:03 +01:00
4d4ee801c0 Guard against strange derivations for packages
Where the requested system doesn't match that of the returned derivation. This
seems to happen for packages like dev86, wine and go.
2020-04-24 21:37:35 +01:00
d09a1ca791 Actually load the (guix lint) module if it's available 2020-04-20 14:29:33 +01:00
780ce66272 Adapt to the different inferior error handling
In recent versions of Guix on the Guix Data Service side, I don't think these
are system-error's, but a different type. For compatibility, just catch any
error.
2020-04-20 08:18:59 +01:00
c4969eb33f Improve detecting if the (guix lint) module is available
Previously it would cause an error when it's not available, which is a bit
unnecessary.
2020-04-20 08:18:19 +01:00
1b197884d0 Print out the system tests as the derivations are being computed
This might help with debugging failures in the load jobs around this point.
2020-04-07 20:08:26 +01:00
cdcc6140aa Guard against errors loading system tests 2020-04-07 18:22:13 +01:00
7db6a8fa03 Guard against errors when inserting log data
PostgreSQL seems to be unhappy with some data, I guess because it gets a bit
jumbled. Rather than failing the job, or getting stuck not inserting logs, try
and capture the error, log the details, and then keep going.
2020-04-07 17:38:57 +01:00
994318782a Pass #:store to checkers that support it
The derivation checker currently opens a store connection on its own, but by
passing the already open connection in, it won't have to do that, and
hopefully this will make checking all the packages faster.
2020-03-24 21:36:13 +00:00
ca0d3ee754 Stop using package_versions_by_guix_revision_range
It's been replaced by the package_derivations_by_guix_revision_range table.
2020-03-24 20:44:57 +00:00
c44297b615 Generate and store system test derivations for all supported systems
Rather than just the native system. I'm not quite sure of the value here, as I
guess system tests should behave the same regardless of the way the software
is compiled, but this seems like it could be useful, and being explicit about
the system the derivation is for is good.
2020-03-20 08:28:55 +00:00
e0f920bb14 Fix marking jobs as failed when exceptions are raised
The switch away from catch broke this, I obviously still don't quite get how
with-exception-handler works. Therefore, use it twice as that seems to help.
2020-03-19 20:43:37 +00:00
185ba36853 Log the time spent in a locked section 2020-02-28 20:41:44 +00:00
f23223f16d Improve the locking around fetching channel instances
I think there's still the potential for a race condition when working with the
cached checkout of the repository, but this lock was far too wide. The
sensitive section is just the latest-channel-instances call, as it adds the
relevant files to the store, and that's what's returned.
2020-02-28 20:40:31 +00:00
a95c5ca5c9 Improve the error reporting within process-load-new-guix-revision-job
By including a backtrace.
2020-02-27 21:14:58 +00:00
b166177dbd Prevent inlining for a number of procedures
So that the load-new-guix-revision tests work with Guile 3.
2020-02-27 21:14:34 +00:00
71f483c816 Add a few imports in preparation for Guile 3 2020-02-27 21:12:16 +00:00
2f41fe79be Adapt some license related code to work without mock in the tests
With Guile 3, there's a potential for mock to work in even fewer
circumstances. So, adapt the code to enable writing the tests without mock.
2020-02-24 21:47:34 +00:00
ce10833459 Switch to using some shorter syntax for logging time taken
In the load-new-guix-revision module.
2020-02-24 21:31:04 +00:00
f57b5c4a9a Add missing ROLLBACK for transaction 2020-02-24 18:50:34 +00:00
cee9acaa87 Refactor the logging setup out of process-load-new-guix-revision-job
To simplify both procedures.
2020-02-24 18:46:53 +00:00
36254f98e3 Improve the job logging
Switch to using a sequence for the ids in the log parts table, and spawn a
thread to listen for output from the inferior processes, and enter it in to
the database.
2020-02-15 17:42:07 +00:00
40f6de27f6 Make deleting the existing log parts optional 2020-02-15 16:49:37 +00:00
46c8ce5b82 Define a new parameter for a port used for the inferior error output 2020-02-15 13:41:17 +00:00
f6f51bbe0d Fake the store connection in the tests
So that the tests don't require a store connection.
2020-02-14 09:18:28 +00:00
5410961f65 Add temporary roots for channel instance derivations 2020-02-13 23:11:36 +00:00
0ef3349ed8 Use a more long lived store connection for loading data
As this will enable registering temporary roots, to avoid store items being
garbage collected.
2020-02-13 23:11:36 +00:00
9be7dbac0b Start storing channel instance derivations
These are the ones that relate to Guix pull.
2020-02-11 08:56:24 +00:00
406aa5e160 Switch using set-current-system to parameterize
As this is neater.
2020-02-11 08:55:34 +00:00
e0ee3d224b Refactor how channel derivations are handled
Yet again...

This makes the channel-derivations for each system accessible within the
load-new-guix-revision procedure, in preparation for storing them in the
database.
2020-02-09 21:40:56 +00:00
60d53f898f Collpase store-item-for-channel in to channel->guix-store-item 2020-02-09 21:06:12 +00:00
13b0ebe561 Start computing channel instance derivations for multiple systems
These aren't stored yet, but this is a start.
2020-02-09 21:03:05 +00:00
beab4babac Further refactor channel->derivation-file-name
To reduce the line length.
2020-02-09 16:44:18 +00:00
9c38115c3d Refactor channel->derivation-file-name to reduce line length 2020-02-09 16:39:46 +00:00