c335775ed4
Check the WAL size before computing lint warnings
...
As even though this might not use the store, the inferior still holds a store
connection open.
2024-08-16 11:18:40 +01:00
eeda1bf33b
Use system-ids for inserting distribution counts
2024-08-12 09:14:36 +01:00
b79c7e83be
Reduce chunk size
...
To try and respond quicker when the WAL is growing.
2024-08-12 09:14:36 +01:00
97b76000ec
Check the WAL size for each chunk
...
Rather than for each system and target, as this should mean the
response (pausing and allowing inferiors and store connections to be closed)
is quicker.
2024-08-12 09:14:36 +01:00
371d76456f
Start trying to handle GC happening while processing revisions
2024-08-12 09:14:36 +01:00
7869082816
Parallelise inserting package derivation distribution counts
2024-08-10 18:28:53 +01:00
885a4758d3
Get the test suite working again
2024-08-07 17:21:49 +01:00
77962f7c2c
Move inserting derivations in to the load-new-guix-revision module
...
And start to more closely integrate it. This makes it possible to start making
it faster by doing more in parallel.
2024-08-07 17:21:28 +01:00
7df7fd3e52
Compute package derivations in chunks
...
This allows for keeping the inferiors and store connections around for a more
constant period, and allows closing the store connections and allowing the
guix-daemon to clear the WAL file if needed.
2024-07-19 19:45:07 +01:00
ed4ba8eb95
Reduce the WAL size threshold
2024-07-19 19:44:53 +01:00
3f1c2ad603
Rewrite the key parts of loading data to be even more parallel
...
Use a pool for the database connection, and a fibers promise for the package
ids, and run other parts of the process in parallel too. This change also
means that inserting starts as soon as some data is available, rather than
when all the data is available.
2024-07-19 19:44:53 +01:00
587277f347
Insert package derivations earlier
2024-07-19 12:15:21 +01:00
2da3a195ea
Return two values from channel->source-and-derivations-by-system
2024-07-19 11:50:14 +01:00
b8d9ed19b2
Avoid long running store connections
...
As I think this can cause the guix-daemon WAL to grow excessively.
2024-07-19 11:47:36 +01:00
0ca9c3f64f
Catch and ignore the new cross build derivation errors
2024-07-18 13:58:21 +01:00
b22834dae7
Refactor opening store connections when processing jobs
...
And set the #:built-in-builders.
2024-07-18 13:57:50 +01:00
7da355b034
Log more timing infromation about inserting derivations
2024-07-16 16:07:47 +01:00
e37eb34db6
Block asyncs when starting inferiors
...
Because this code deals with global state, like environment variables.
2024-06-24 23:02:41 +01:00
31bd2156f7
Support setting environment variables in the inferior
...
When processing jobs, this is mostly to allow setting GUIX_DOWNLOAD_METHODS.
2024-06-24 23:02:14 +01:00
2043a4ef6f
Fix WAL threshold
...
As it was too small.
2024-05-13 17:20:25 +01:00
824e731a5c
Attempt to avoid the WAL growing excessively
2024-04-25 23:33:12 +01:00
4a09be753c
Use non-blocking inferior-eval-with-store in for glibc-locales
2024-04-25 23:29:50 +01:00
993887fe0c
Further change job store connection handling
...
The guix-dameon WAL is still growing excessively, so avoid doing anything with
the long running store connection except registering temporary roots.
2024-04-25 22:01:22 +01:00
88b9d34fb1
Close store connections earlier
...
As I think the temporary roots on the long running store connection should be
sufficient.
2024-04-15 13:43:37 +01:00
5f5b8fc91d
Stop using a inferior-package-derivations lock
...
As this isn't for correctness reasons, but resource usage. I'm hoping to
manage this differently.
2024-03-31 19:55:46 +01:00
b4583bb177
Fallback when building channel instances
2024-03-13 15:34:48 +00:00
7d1cc4d325
Attempt to avoid issues with the guix-daemon WAL
2024-03-08 11:29:00 +00:00
5f80856ad1
Allow prioritising jobs from specific repositories
...
To help the QA data service prioritise branches over patches.
2024-03-07 10:02:18 +00:00
20ad81f0d4
Further fix handling exceptions when computing channel instances
2024-02-13 10:35:04 +00:00
26f517d9c2
Improve exception handling for channel instance derivations
2024-02-03 11:08:04 +01:00
ac1a4cb1e2
Cleanup some with-time-logging
2024-02-02 16:58:06 +01:00
7ecb13e403
Compute lint warnings in parallel
2024-02-01 13:16:08 +01:00
215a67238a
Ignore inferior-protocol-error when computing channel instances
...
Since this seems to happen for i586-gnu for core-updates currently and I can't
seem to reproduce the issue locally or work out what's wrong.
2024-02-01 10:47:54 +01:00
7acb38377e
Fix exception handling in call-with-temporary-thread
2024-02-01 10:39:11 +01:00
9779ecf3cd
Speed up fetching package replacements
2024-01-31 12:30:51 +00:00
7d9704192d
Add more logging in when computing channel instance derivations
...
As there's an issue with current core-updates that I'm struggling to track
down.
2024-01-31 08:59:47 +00:00
ba97ade781
Add some time logging in to inferior-packages-plus-replacements
2024-01-28 09:10:01 +00:00
0f7b31c605
Make sure to keep roots for channel instance derivations
2024-01-28 08:18:44 +00:00
75f27264a6
Improve logging when computing a channel instance derivation fails
2024-01-27 18:41:02 +00:00
062ee1aed2
Log the time to call inferior-packages-plus-replacements
...
As I think this is currently quite slow.
2024-01-27 18:40:01 +00:00
e0a6c845e5
Try to fix issues with derivations being GC'ed
2024-01-19 09:57:19 +00:00
c1d2f3a1b7
Add meaningful parallelism to processing jobs
...
Make parallel use of inferiors when computing channel instance derivations,
and when extracting information about a revision. This should allow for some
horizontal scalability, reducing the impact of additional systems for which
derivations need computing.
This commit also fixes an apparent issue with package replacements, as
previously the wrong id was used, and this hid some issues around
deduplication.
2024-01-18 15:34:40 +00:00
45d9b1ad7c
Add back inferior heap size reporting
...
I'm sure this was present before, but maybe lost during some refactoring.
2024-01-18 14:41:32 +00:00
241d7e4889
Use libc-locales-for-target
...
To help find the right glibc-locales to use.
2023-12-04 11:11:39 +00:00
b2707b4828
Restore #:error-port when calling open-inferior
...
As the default is a void port.
2023-11-05 16:12:55 +00:00
1746256ef5
Remove #:use-module (ice-9 custom-ports)
...
As it's not actually used.
2023-11-05 15:59:53 +00:00
c3cb04cb80
Use fibers when processing new revisions
...
Just have one fiber at the moment, but this will enable using fibers for
parallelism in the future.
Fibers seemed to cause problems with the logging setup, which was a bit odd in
the first place. So move logging to the parent process which is better anyway.
2023-11-05 13:46:20 +00:00
f5acc60288
Make some sweeping changes to loading new revisions
...
Move in the direction of being able to run multiple inferior REPLs, and use
some vectors rather than lists in places (maybe this is more efficient).
2023-11-02 12:16:17 +00:00
10bad53ad5
Support polling git repositories for new branches/revisions
...
This is mostly a workaround for the occasional problems with the guix-commits
mailing list, as it can break and then the data service doesn't learn about
new revisions until the problem is fixed.
I think it's still a generally good feature though, and allows deploying the
data service without it consuming emails to learn about new revisions, and is
a step towards integrating some kind of way of notifying the data service to
poll.
2023-10-09 22:19:02 +01:00
abe50b0443
Fix the load revision system sorting
...
As previously it only applied to system tests.
2023-07-25 17:58:19 +01:00
