f5acc60288
Make some sweeping changes to loading new revisions
...
Move in the direction of being able to run multiple inferior REPLs, and use
some vectors rather than lists in places (maybe this is more efficient).
2023-11-02 12:16:17 +00:00
89782b3449
Close the repository when pulling
...
And remove the monitor, as I'm not sure it's necessary.
2023-10-16 21:18:44 +01:00
851a6c17b5
Remove redundant joins from the select build query
2023-10-16 17:19:44 +01:00
82bb174700
Attempt to fetch this Git polling mess
...
Listing remote branches through libgit2 will list branches that don't exist on
the remote. I think branch-list is more listing branch references, and you can
have references to a remote branch where the remote branch doesn't exist. This
isn't very useful here though, as I'm trying to work out what remote branches
exist.
There's remote-ls which might help, but I can't figure out how to get the
commits for branches from that.
Therefore, just bodge the two things together in to a big mess. I seem to be
able to get commits from branch-list that hopefully match what's on the
remote (although I'm not confident about this), and I think remote-ls does
allow checking what branches exist.
2023-10-11 17:53:38 +01:00
70f1824e46
Copy some of the initialisation from (guix git)
...
To go along with just-update-cached-checkout.
2023-10-11 16:33:53 +01:00
695fce6922
Try to work around update-cached-checkout
...
As it's causing problems with the guix-patches repository.
2023-10-11 16:26:08 +01:00
d72b9a5239
Remove logging from enqueue-job-for-email
...
As getmail treats output to stderr as the operation failing.
2023-10-11 14:17:11 +01:00
b0404d62fd
Fix typo
2023-10-11 14:04:40 +01:00
10bad53ad5
Support polling git repositories for new branches/revisions
...
This is mostly a workaround for the occasional problems with the guix-commits
mailing list, as it can break and then the data service doesn't learn about
new revisions until the problem is fixed.
I think it's still a generally good feature though, and allows deploying the
data service without it consuming emails to learn about new revisions, and is
a step towards integrating some kind of way of notifying the data service to
poll.
2023-10-09 22:19:02 +01:00
9bb8f84741
Add commit message entry about UDP push notifications for commits
2023-09-27 15:36:00 +01:00
893cccf8e5
Add metrics for ports and file descriptors
2023-09-07 11:16:45 +01:00
e8e1459769
URI encode some derivation names in links
...
To handle characters like +.
2023-09-07 11:14:11 +01:00
f2e2ca7a07
Use the connection pool for build events
...
I'm not sure why this wasn't the way it worked previously. This should avoid
unconstrained use of PostgreSQL connections.
2023-08-23 12:19:22 +01:00
9f102dbd39
Add code to delete nars entries
2023-08-01 14:13:10 +01:00
1461aa037f
Make database changes to help with deleting entries in the nars table
2023-08-01 14:12:48 +01:00
7495085f63
Delete unreferenced derivations in batches
...
To avoid a long blocking query.
2023-08-01 10:16:31 +01:00
1c75394187
Fix issues with with-resource-from-pool
2023-07-26 10:11:00 +01:00
abe50b0443
Fix the load revision system sorting
...
As previously it only applied to system tests.
2023-07-25 17:58:19 +01:00
bbc53deb1f
Rewrite deleting unreferenced derivations
...
Use fibers more, leaning in on the non-blocking use of Squee for parallelism.
2023-07-25 17:57:00 +01:00
348fe36b55
Sort systems and targets for more deterministic behaviour
...
As the ordering from Guix seems to be non-deterministic.
2023-07-21 12:10:55 +01:00
ef73305250
Further tweak loading package derivations
...
There's an issue where sometimes for i686-linux and armhf-linux, only a few
package derivations are computed.
This commit tries to simplify the code, and adds some conditional logging for
the guix package, which might help reveal what's going on.
2023-07-20 16:46:53 +01:00
6e4d436d75
Fix some uses of select-build-server-urls-by-id
2023-07-20 16:12:00 +01:00
ce1480edf4
Fix the repositories page
2023-07-19 17:40:41 +01:00
de6d38016d
Tweak which duplicate packages to select
...
This helps with replacements, as the original package is usually higher up in
the file.
2023-07-17 13:28:42 +01:00
ed974ebf3b
Tweak loading package derivations
...
Make sure to log any errors, and also use a more efficient approach sending
less data to the inferior.
2023-07-17 13:27:21 +01:00
9dec45d2eb
Make fetching metrics work even when having database problems
2023-07-14 10:36:02 +01:00
f9bb60ab4a
Drop the resource pool timeout
...
Better to timeout early.
2023-07-14 10:35:44 +01:00
22ed81fbcc
Reduce the number of threads used by fibers
...
As one thread per core is probably unnecessary.
2023-07-10 18:56:31 +01:00
2971322b27
Set thread names
...
To help with debugging
2023-07-10 18:56:31 +01:00
75ef4cffd3
Expose resource pool stats
2023-07-10 18:56:31 +01:00
899bd1387e
Support getting resource pool stats
2023-07-10 18:56:31 +01:00
7251c7d653
Stop using a pool of threads for database operations
...
Now that squee cooperates with suspendable ports, this is unnecessary. Use a
connection pool to still support running queries in parallel using multiple
connections.
2023-07-10 18:56:31 +01:00
672ee6216e
Try to fix backfilling blocked_builds
2023-07-02 10:42:49 +01:00
d12ae0079e
Filter out duplicate ids for blocking builds
2023-07-02 10:42:24 +01:00
742949cc97
Improve data deletion
2023-07-01 12:01:13 +01:00
044a905c1d
Provide more information about revision processing errors
...
In the compare package derivations response.
2023-06-15 11:36:46 +01:00
f7b823cfd9
Fix fetch-narinfo-files to work in the specific outputs case
2023-06-09 16:32:54 +01:00
04e90a17b0
Better report exceptions in query-build-server-substitutes
2023-06-09 16:11:06 +01:00
29d49ba31a
Detach the database setup from the main guix-data-service process
...
This will allow restarting them independently, leaving it up to the operator
to ensure that all processes are compatible.
2023-06-09 16:11:06 +01:00
5c9ec28cb5
Query for outputs when build events arrive
...
This will keep the substitute information more up to date.
2023-06-09 16:11:06 +01:00
7524d23b44
Make the build event handling code more generic
...
So that triggering a check for substitutes can be integrated in.
2023-06-06 12:32:53 +01:00
68850065d7
Extend the invalid parameter handling for more types
2023-05-22 17:34:39 +01:00
5f21458a8f
Have render-compare/package-derivations provide invalid param info
...
So that this can be used by the qa-frontpage.
This should be improved and generalised.
2023-05-22 17:25:44 +01:00
54c7a1a880
Fix ignoring canceled builds
...
The previous changes only affected searching for package derivations, and they
also didn't work.
2023-05-18 12:31:58 +01:00
4208b5f148
Ignore canceled builds when querying package derivations
...
This will help when using this to submit builds, since you won't end up
ignoring derivations with canceled builds.
2023-05-18 11:25:14 +01:00
61f6619313
Add Guile internal time metrics
...
As this will help identify when the service restarts.
2023-05-14 22:45:19 +01:00
bbacbcd2aa
Ensure the known and unknown keys appear
2023-05-09 13:45:36 +01:00
c50f6aeb34
Remove redundant match-lambda in select-package-output-availability-for-revision
2023-05-09 13:43:32 +01:00
47c482bdcc
Set lock_timeout for some data deletion transactions
...
As these can cause deadlocks. This will probably cause errors, so some
retrying will need to be added.
2023-05-09 08:55:09 +01:00
3734a85650
Use the package_derivations system id in a query
...
Rather than the derivations system id, as this helps PostgreSQL run the query
faster.
2023-05-04 08:32:43 +01:00
