f2e2ca7a07
Use the connection pool for build events
...
I'm not sure why this wasn't the way it worked previously. This should avoid
unconstrained use of PostgreSQL connections.
2023-08-23 12:19:22 +01:00
9f102dbd39
Add code to delete nars entries
2023-08-01 14:13:10 +01:00
1461aa037f
Make database changes to help with deleting entries in the nars table
2023-08-01 14:12:48 +01:00
7495085f63
Delete unreferenced derivations in batches
...
To avoid a long blocking query.
2023-08-01 10:16:31 +01:00
1c75394187
Fix issues with with-resource-from-pool
2023-07-26 10:11:00 +01:00
abe50b0443
Fix the load revision system sorting
...
As previously it only applied to system tests.
2023-07-25 17:58:19 +01:00
bbc53deb1f
Rewrite deleting unreferenced derivations
...
Use fibers more, leaning in on the non-blocking use of Squee for parallelism.
2023-07-25 17:57:00 +01:00
348fe36b55
Sort systems and targets for more deterministic behaviour
...
As the ordering from Guix seems to be non-deterministic.
2023-07-21 12:10:55 +01:00
ef73305250
Further tweak loading package derivations
...
There's an issue where sometimes for i686-linux and armhf-linux, only a few
package derivations are computed.
This commit tries to simplify the code, and adds some conditional logging for
the guix package, which might help reveal what's going on.
2023-07-20 16:46:53 +01:00
6e4d436d75
Fix some uses of select-build-server-urls-by-id
2023-07-20 16:12:00 +01:00
ce1480edf4
Fix the repositories page
2023-07-19 17:40:41 +01:00
de6d38016d
Tweak which duplicate packages to select
...
This helps with replacements, as the original package is usually higher up in
the file.
2023-07-17 13:28:42 +01:00
ed974ebf3b
Tweak loading package derivations
...
Make sure to log any errors, and also use a more efficient approach sending
less data to the inferior.
2023-07-17 13:27:21 +01:00
9dec45d2eb
Make fetching metrics work even when having database problems
2023-07-14 10:36:02 +01:00
f9bb60ab4a
Drop the resource pool timeout
...
Better to timeout early.
2023-07-14 10:35:44 +01:00
22ed81fbcc
Reduce the number of threads used by fibers
...
As one thread per core is probably unnecessary.
2023-07-10 18:56:31 +01:00
2971322b27
Set thread names
...
To help with debugging
2023-07-10 18:56:31 +01:00
75ef4cffd3
Expose resource pool stats
2023-07-10 18:56:31 +01:00
899bd1387e
Support getting resource pool stats
2023-07-10 18:56:31 +01:00
7251c7d653
Stop using a pool of threads for database operations
...
Now that squee cooperates with suspendable ports, this is unnecessary. Use a
connection pool to still support running queries in parallel using multiple
connections.
2023-07-10 18:56:31 +01:00
672ee6216e
Try to fix backfilling blocked_builds
2023-07-02 10:42:49 +01:00
d12ae0079e
Filter out duplicate ids for blocking builds
2023-07-02 10:42:24 +01:00
742949cc97
Improve data deletion
2023-07-01 12:01:13 +01:00
044a905c1d
Provide more information about revision processing errors
...
In the compare package derivations response.
2023-06-15 11:36:46 +01:00
f7b823cfd9
Fix fetch-narinfo-files to work in the specific outputs case
2023-06-09 16:32:54 +01:00
04e90a17b0
Better report exceptions in query-build-server-substitutes
2023-06-09 16:11:06 +01:00
29d49ba31a
Detach the database setup from the main guix-data-service process
...
This will allow restarting them independently, leaving it up to the operator
to ensure that all processes are compatible.
2023-06-09 16:11:06 +01:00
5c9ec28cb5
Query for outputs when build events arrive
...
This will keep the substitute information more up to date.
2023-06-09 16:11:06 +01:00
7524d23b44
Make the build event handling code more generic
...
So that triggering a check for substitutes can be integrated in.
2023-06-06 12:32:53 +01:00
68850065d7
Extend the invalid parameter handling for more types
2023-05-22 17:34:39 +01:00
5f21458a8f
Have render-compare/package-derivations provide invalid param info
...
So that this can be used by the qa-frontpage.
This should be improved and generalised.
2023-05-22 17:25:44 +01:00
54c7a1a880
Fix ignoring canceled builds
...
The previous changes only affected searching for package derivations, and they
also didn't work.
2023-05-18 12:31:58 +01:00
4208b5f148
Ignore canceled builds when querying package derivations
...
This will help when using this to submit builds, since you won't end up
ignoring derivations with canceled builds.
2023-05-18 11:25:14 +01:00
61f6619313
Add Guile internal time metrics
...
As this will help identify when the service restarts.
2023-05-14 22:45:19 +01:00
bbacbcd2aa
Ensure the known and unknown keys appear
2023-05-09 13:45:36 +01:00
c50f6aeb34
Remove redundant match-lambda in select-package-output-availability-for-revision
2023-05-09 13:43:32 +01:00
47c482bdcc
Set lock_timeout for some data deletion transactions
...
As these can cause deadlocks. This will probably cause errors, so some
retrying will need to be added.
2023-05-09 08:55:09 +01:00
3734a85650
Use the package_derivations system id in a query
...
Rather than the derivations system id, as this helps PostgreSQL run the query
faster.
2023-05-04 08:32:43 +01:00
639c6ff183
Further tweak fetching narinfos
...
Move the batching to the database, which should reduce memory usage while
removing the limit on the number of fetched narinfos.
2023-04-28 22:33:41 +02:00
8a19bcba13
Tweak fetching narinfos
...
Currently I'm seeing failures due to guile-gnutls not supporting suspendable
ports (write_wait_fd), so batch the requested outputs to try and avoid this.
2023-04-28 20:42:31 +02:00
688f4cd79d
Set request timeouts for the thread pools
...
The request timeout should ensure that the operations don't back up if the
thread pool is overloaded.
2023-04-27 14:58:47 +02:00
638e0442c3
Support request timeouts in the thread pool
2023-04-27 14:58:44 +02:00
5bb7cf0c1c
Change par-mapper' to use fetch-result-of-defered-thunks
...
So the fetch-result-of-defered-thunk procedure can be removed.
2023-04-27 11:59:01 +02:00
9f080524bc
Split the thread pool used for database connections
...
In to two thread pools, a default one, and one reserved for essential
functionality.
There are some pages that use slow queries, so this should help stop those
pages block other operations.
2023-04-27 10:31:09 +02:00
4fa7a3601e
Include distribution counts table in data deletion
2023-04-07 11:21:28 +01:00
1a0eaeb672
Improve performance of select-fixed-output-package-derivations-in-revision
2023-03-11 18:19:19 +00:00
a20b929e90
Fix query in get-count-for-next-level
2023-03-09 14:05:41 +00:00
519f0c6f67
Defer backfilling derivation distribution counts until later
...
After the migrations have run.
2023-03-09 09:39:47 +00:00
319439d05a
Reorder some vacuum operations when loading revisions
2023-03-09 09:17:07 +00:00
659dcc6a0d
Avoid a recursive CTE for finding blocked builds where possible
...
Use the new approach of looking up the distribution of the derivations, and
building a non recursive query specifically for this revision. This should
avoid PostgreSQL picking a poor plan for performing the query.
2023-03-09 08:36:51 +00:00
