6808532338
Close unused PostgreSQL connections
...
As they take up memory, so free some by closing them.
2024-04-01 22:20:46 +01:00
9fcd615c1f
Remove the per-thread PostgreSQL connection code
...
As this has been replaced by a fibers resource pool.
2024-04-01 22:20:00 +01:00
b5f59189e1
Move backfilling in to the server module and use the connection pool
...
To avoid using the old PostgreSQL connection per thread code.
2024-04-01 21:51:29 +01:00
5f5b8fc91d
Stop using a inferior-package-derivations lock
...
As this isn't for correctness reasons, but resource usage. I'm hoping to
manage this differently.
2024-03-31 19:55:46 +01:00
5d0e84bac4
Make changes for the guile-prometheus update
2024-03-30 21:37:16 +00:00
b4583bb177
Fallback when building channel instances
2024-03-13 15:34:48 +00:00
a667db2f5d
Don't start new jobs when there's low disk space
2024-03-11 19:34:20 +00:00
7d1cc4d325
Attempt to avoid issues with the guix-daemon WAL
2024-03-08 11:29:00 +00:00
5f80856ad1
Allow prioritising jobs from specific repositories
...
To help the QA data service prioritise branches over patches.
2024-03-07 10:02:18 +00:00
184d341671
Report the size of the guix database and WAL file
...
Since the Guix daemon doesn't export these metrics, instrument it here.
2024-03-05 11:15:04 +00:00
ca69d3329d
Add exception handling to the process-jobs script
...
As I'm seeing this exit on beid, but I'm not sure why.
2024-03-05 10:57:41 +00:00
20ad81f0d4
Further fix handling exceptions when computing channel instances
2024-02-13 10:35:04 +00:00
26a751570c
Copy the port monitoring fiber from the build coordinator
...
As the data service has the same issue where it stops listening on the port.
2024-02-13 09:59:39 +00:00
76712e2b00
Instrument resource pool checkout failures
...
As I've got no idea why the resource pools sometimes stop working.
2024-02-10 10:11:29 +00:00
26f517d9c2
Improve exception handling for channel instance derivations
2024-02-03 11:08:04 +01:00
ac1a4cb1e2
Cleanup some with-time-logging
2024-02-02 16:58:06 +01:00
7ecb13e403
Compute lint warnings in parallel
2024-02-01 13:16:08 +01:00
0468bc4321
Speed up loading package metadata
...
By batching the SQL queries.
2024-02-01 13:15:30 +01:00
215a67238a
Ignore inferior-protocol-error when computing channel instances
...
Since this seems to happen for i586-gnu for core-updates currently and I can't
seem to reproduce the issue locally or work out what's wrong.
2024-02-01 10:47:54 +01:00
7acb38377e
Fix exception handling in call-with-temporary-thread
2024-02-01 10:39:11 +01:00
d5e663cc1e
Split up handling of package description data
...
To hopefully see which part is slow.
2024-01-31 20:20:48 +01:00
9779ecf3cd
Speed up fetching package replacements
2024-01-31 12:30:51 +00:00
7d9704192d
Add more logging in when computing channel instance derivations
...
As there's an issue with current core-updates that I'm struggling to track
down.
2024-01-31 08:59:47 +00:00
ba97ade781
Add some time logging in to inferior-packages-plus-replacements
2024-01-28 09:10:01 +00:00
0f7b31c605
Make sure to keep roots for channel instance derivations
2024-01-28 08:18:44 +00:00
39f626aa45
Remove even more time logging
2024-01-28 08:18:13 +00:00
75f27264a6
Improve logging when computing a channel instance derivation fails
2024-01-27 18:41:02 +00:00
062ee1aed2
Log the time to call inferior-packages-plus-replacements
...
As I think this is currently quite slow.
2024-01-27 18:40:01 +00:00
e51d2f8932
Remove some time logging
...
As this is a bit noisy.
2024-01-27 18:39:41 +00:00
e0a6c845e5
Try to fix issues with derivations being GC'ed
2024-01-19 09:57:19 +00:00
a900a1c2ec
Remove drain? #t from process job
...
As it now uses more fibers.
2024-01-18 22:41:02 +00:00
b81e7ea950
Fixup tests
2024-01-18 15:34:40 +00:00
c1d2f3a1b7
Add meaningful parallelism to processing jobs
...
Make parallel use of inferiors when computing channel instance derivations,
and when extracting information about a revision. This should allow for some
horizontal scalability, reducing the impact of additional systems for which
derivations need computing.
This commit also fixes an apparent issue with package replacements, as
previously the wrong id was used, and this hid some issues around
deduplication.
2024-01-18 15:34:40 +00:00
6842a432d6
Split and instrument parts of inferior-packages->package-metadata-ids
...
As parts of it are slow.
2024-01-18 14:42:48 +00:00
e51f87cc7e
Rewrite part of insert-missing-data-and-return-all-ids to avoid filter
...
As filter can use part of the input list, which then prevents modifying the
filtered list.
2024-01-18 14:41:54 +00:00
b8bd56a8f5
Fix par-map&
...
To actually work in parallel.
2024-01-18 14:41:32 +00:00
46b4c87e92
Show backtraces when using parallel fibers and resource pools
2024-01-18 14:41:32 +00:00
5af6233e5b
Make it possible to destroy a resource pool
...
And implement removing idle resources.
2024-01-18 14:41:32 +00:00
15b6dad5a5
Have delete-duplicates/sort! take a equality procedure
...
And change the default, as eq? doesn't always work.
2024-01-18 14:41:32 +00:00
45d9b1ad7c
Add back inferior heap size reporting
...
I'm sure this was present before, but maybe lost during some refactoring.
2024-01-18 14:41:32 +00:00
60980e2668
Use delete-duplicates/sort! in inferior-packages->license-set-ids
...
As it should offer a speedup over delete-duplicates.
2024-01-18 14:41:32 +00:00
49b4841c4e
Use delete-duplicates/sort! in insert-missing-data-and-return-all-ids
...
As it's faster than delete-duplicates for large amounts of data.
2024-01-18 14:41:32 +00:00
241d7e4889
Use libc-locales-for-target
...
To help find the right glibc-locales to use.
2023-12-04 11:11:39 +00:00
8a8dccf2d8
Fix some select-build-servers calls
2023-12-04 11:09:46 +00:00
a3ec1f326d
Set %file-port-name-canonicalization when processing jobs
...
Just in case this helps with performance.
2023-12-04 11:06:27 +00:00
b9026488ed
Memoize computing tokens
...
As I'm not sure how expensive this is, but it doesn't need doing for every
request.
2023-11-24 16:11:57 +00:00
241a704db1
Instrument handling build events
2023-11-24 16:11:42 +00:00
e13febc817
Stop replacements cluttering comparisons
...
This stops the replaced package getting compared with it's replacement.
2023-11-18 10:36:10 +00:00
b2bf948a00
Add more exception handling to make-resource-pool
...
As I'm not sure it's working reliably.
2023-11-17 12:32:14 +00:00
da2a405e8b
Remove hardcoding of builtin:download
...
As this was causing problems with builtin:git-download.
2023-11-16 12:20:55 +00:00
