77962f7c2c
Move inserting derivations in to the load-new-guix-revision module
...
And start to more closely integrate it. This makes it possible to start making
it faster by doing more in parallel.
2024-08-07 17:21:28 +01:00
7f746b358b
Add the fiberize utility
2024-08-07 16:50:30 +01:00
3d2335cebe
Stream the render-revision-package-derivations JSON response
...
As it can be large.
2024-08-05 13:45:46 +01:00
d8ade0ea11
Fix the pool metrics
2024-07-29 11:24:24 +01:00
f0f1329461
Tweak timeouts in resource-pool-stats
2024-07-29 11:24:08 +01:00
fbbdc41f5c
Update tests
2024-07-26 22:21:59 +01:00
e8bd27fdb8
Try using 2 threads for the server
2024-07-26 21:42:21 +01:00
2a80304e0c
Add worker thread utils
2024-07-20 00:20:20 +01:00
7df7fd3e52
Compute package derivations in chunks
...
This allows for keeping the inferiors and store connections around for a more
constant period, and allows closing the store connections and allowing the
guix-daemon to clear the WAL file if needed.
2024-07-19 19:45:07 +01:00
ed4ba8eb95
Reduce the WAL size threshold
2024-07-19 19:44:53 +01:00
3f1c2ad603
Rewrite the key parts of loading data to be even more parallel
...
Use a pool for the database connection, and a fibers promise for the package
ids, and run other parts of the process in parallel too. This change also
means that inserting starts as soon as some data is available, rather than
when all the data is available.
2024-07-19 19:44:53 +01:00
5439159a16
Add more fibers utilities
2024-07-19 19:44:53 +01:00
587277f347
Insert package derivations earlier
2024-07-19 12:15:21 +01:00
2da3a195ea
Return two values from channel->source-and-derivations-by-system
2024-07-19 11:50:14 +01:00
b8d9ed19b2
Avoid long running store connections
...
As I think this can cause the guix-daemon WAL to grow excessively.
2024-07-19 11:47:36 +01:00
0ca9c3f64f
Catch and ignore the new cross build derivation errors
2024-07-18 13:58:21 +01:00
b22834dae7
Refactor opening store connections when processing jobs
...
And set the #:built-in-builders.
2024-07-18 13:57:50 +01:00
e81c6377bf
Re-work the fibers scheduling
...
Use a single thread for receiving and responding to requests, and delegate the
processing of the requests to a separate set of threads.
I'm hoping this will avoid the processing of requests affecting accepting new
connections, or the sending of responses.
2024-07-16 23:03:15 +01:00
d8e9de4ed6
Use a bigger buffer for requests/responses
...
As I'm hoping this will increase the data throughput for responses.
2024-07-16 20:37:07 +01:00
5f8cd2dc56
Stream the package derivation page since it can be large
2024-07-16 17:16:22 +01:00
bbbcea8ff6
Add more time logging in to insert-missing-derivations
2024-07-16 16:13:17 +01:00
7da355b034
Log more timing infromation about inserting derivations
2024-07-16 16:07:47 +01:00
1754d1a321
Stop inserting missing source file nars
...
This was more an issue several years ago, so this code is not really needed
now.
2024-07-16 16:06:46 +01:00
a61c4baccd
Try and speed up large package derivation comparisions
2024-07-12 13:33:37 +01:00
9032079bda
Fix delete-revisions-from-branch
2024-07-07 11:59:53 +02:00
de98cbfaa3
Remove the statistics page
...
As it's been broken for a long time now, there are too many rows to count.
2024-06-30 11:14:16 +01:00
335715bc72
Properly handle deleting commits that occur on multiple branches
2024-06-29 11:49:45 +01:00
b5fbde5ac8
Delete duplicates when inserting license data
...
As I think this is necessary.
2024-06-25 14:41:16 +01:00
e37eb34db6
Block asyncs when starting inferiors
...
Because this code deals with global state, like environment variables.
2024-06-24 23:02:41 +01:00
31bd2156f7
Support setting environment variables in the inferior
...
When processing jobs, this is mostly to allow setting GUIX_DOWNLOAD_METHODS.
2024-06-24 23:02:14 +01:00
d7103eccc9
Fix the arguments to most-appropriate-mime-type
2024-06-24 09:59:37 +01:00
eeb026b654
Further tweak the package derivation comparison query
2024-06-22 08:55:44 +01:00
e6205e988a
Speed up querying for revision package derivations
...
By splitting it up by system.
2024-06-21 15:29:34 +01:00
4e7c2bcfbf
Don't compare across systems in one query
...
As the query seems to be super slow, and this allows parallelising it as well.
2024-06-21 12:11:48 +01:00
94e66d5b1f
Add a JSON error page
2024-06-21 11:04:57 +01:00
84a2ad5b25
Speed up deleting derivation sources
2024-06-20 17:07:41 +01:00
530f58b59c
Cache the derivations that weren't deleted
...
When deleting derivations, as I think this might reduce the number of queries.
2024-06-20 15:47:21 +01:00
5e88a98c47
Speed up select-build-outputs
2024-06-20 15:33:29 +01:00
a9dbeb5e09
Reduce max-age for the latest-processed-commit page
...
As this shouldn't be cached for a day.
2024-06-12 16:54:35 +01:00
d74422c268
Add more logging around polling git repositories
2024-05-23 09:38:24 +01:00
f7ada4bf1f
Guard against trying to delete an empty list of commits
2024-05-22 11:46:18 +01:00
5d50a0e3e1
Support regexes for included and excluded branches
2024-05-22 11:00:04 +01:00
2043a4ef6f
Fix WAL threshold
...
As it was too small.
2024-05-13 17:20:25 +01:00
f4be647610
Use a separate fiber to send pool stats
...
As the requester might timeout.
2024-05-02 20:41:04 +01:00
957727c51a
Fix package replacement handling on the revision packages page
2024-04-28 22:04:08 +01:00
6201099003
Instrument requests
2024-04-26 09:46:10 +01:00
824e731a5c
Attempt to avoid the WAL growing excessively
2024-04-25 23:33:12 +01:00
4a09be753c
Use non-blocking inferior-eval-with-store in for glibc-locales
2024-04-25 23:29:50 +01:00
993887fe0c
Further change job store connection handling
...
The guix-dameon WAL is still growing excessively, so avoid doing anything with
the long running store connection except registering temporary roots.
2024-04-25 22:01:22 +01:00
c18589249f
Instrument the resource pool checkout timeouts
2024-04-15 13:54:25 +01:00
