cc61bb5f13
Drop the chunk size when gathering lint warnings
...
To try and bring the peak memory usage down.
2022-11-14 09:26:59 +00:00
ab7df4c6e5
Include blocked_builds information in comparison responses
...
This will make it easier to tell when a scheduled build is yet to start, and
can't start due to a missing dependency.
2022-11-14 09:24:49 +00:00
8294accffe
Remove Build status field from blocking builds page
...
As this is unused.
2022-11-12 11:54:50 +00:00
c46ee47632
Make backfilling blocked_builds a bit smarter
...
And drop the chunk size.
2022-11-12 11:53:14 +00:00
ed114265cd
Handle deleting from blocked_builds when builds are scheduled
...
As scheduling a build might unblock others.
2022-11-12 11:42:26 +00:00
b9305d81a4
View scheduled builds like succeeded builds in terms of blocking
...
This means that an output is viewed to not be blocking if it has a scheduled
build, just as if it has a succeeded build. Also, scheduling builds will
unblock blocked builds.
This is helpful as it means that it reduces noise for blocking builds.
2022-11-12 11:33:37 +00:00
7731c6f340
Tweak backfilling the blocked builds
2022-11-12 10:57:53 +00:00
8e9ab68d14
Use latest_build_status rather than build_status
...
In various places in the blocked-builds module.
2022-11-12 10:57:26 +00:00
a34bf4defc
Spawn specific PostgreSQL connections for the blocked_builds updates
...
So that the queries don't get cancelled by the statement timeout.
2022-11-12 10:46:46 +00:00
fc5f562731
Add index on derivation_outputs id and derivation_id fields
...
This might help with doing recursive queries on the derivations graph.
2022-11-12 10:42:04 +00:00
fb9d99a076
Add extended statistics on package_derivations
...
This helps row count estimates when filtering on system_id and target.
2022-11-12 10:40:43 +00:00
48d8ee885a
Have insert-blocked-builds cache when the partitions exist
...
To make it more efficient.
2022-11-11 11:29:45 +00:00
0f22e3ab40
Rework insert-blocked-builds to make it more efficient
...
This also fixes a typo in the partition name.
2022-11-11 11:29:37 +00:00
4101c6b897
Stop using exception handling when inserting blocked_builds entries
...
As it doesn't work in a transaction.
2022-11-11 10:51:12 +00:00
01e54c36db
Add missing test description
2022-11-11 10:35:09 +00:00
989916b740
Add a blocking builds page
2022-11-11 10:35:09 +00:00
1fb291be40
Add support for incrementally tracking blocked builds
...
This will hopefully provide a less expensive way of finding out if a scheduled
build is probably blocked by other builds failing or being canceled.
By working this out when the build events are recieved, it should be more
feasible to include information about whether builds are likely blocked or not
in various places (e.g. revision comparisons).
2022-11-11 10:35:09 +00:00
95064d39a3
Log heap size when computing system tests
2022-11-06 08:53:04 +01:00
1e2826e095
Add more granular logging for computing system test derivations
2022-11-06 08:47:58 +01:00
d06230fcf4
Close postgresql connections when the thread pool thread is idle
...
I think the idle connections associated with idle threads are still taking up
memory, so especially now that you can configure an arbitrary number of
threads (and thus connections), I think it's good to close them regularly.
2022-10-23 11:28:37 +01:00
aaec813cba
Fix JSON response for unknown revisions
2022-10-17 16:07:46 +01:00
8dd164665a
Expose Guile GC metrics
...
As the guix-data-service process seems to be using excessive amounts of
memory, and this will be useful to track it.
2022-10-09 11:13:28 +01:00
ff77bbea7e
Make it possible to increase the number of thread pool threads
...
And double the default to 16.
2022-10-02 15:08:18 +01:00
ce2e13aa45
Log delays in the thread pool channel
...
As I think with lots of requests, this could become a bottleneck.
2022-10-02 14:57:02 +01:00
640386a84d
Insert guix revision lint warnings in chunks
...
To avoid long running queries.
2022-09-17 08:53:23 +02:00
78a5abee21
Improve chunking when inserting derivation inputs
...
Chunk the values inserted in the query, rather than the derivations involved,
as this is more consistent.
2022-09-17 08:53:23 +02:00
7050ea749f
Reduce some chunk sizes
2022-09-17 00:40:51 +02:00
35cf9ba1bc
Chunk inserting guix revision package derivation entries
2022-09-15 16:25:41 +02:00
8ef896b103
Further reduce some chunk sizes
2022-09-15 16:25:31 +02:00
37e8af60fb
Chunk the data for some queries in insert-missing-data-and-return-all-ids
...
This helps to avoid queries getting logged as slow just because of the amount
of data.
2022-09-15 09:18:19 +01:00
d791c2bd8a
Format some queries generated in insert-missing-data-and-return-all-ids
2022-09-14 15:42:16 +01:00
f41bfcf8b6
Reduce some chunk sizes
...
As these queries are still slow enough to be logged.
2022-09-14 15:42:00 +01:00
189b9a4e86
Implement the JSON response for package reproducibility
2022-09-14 10:52:04 +01:00
ec3dba4894
Implement the JSON response for package substitute availability
2022-09-14 10:37:33 +01:00
e3062abf33
Speed up finding the locales for a revision
2022-09-14 10:26:03 +01:00
12af30c039
Reduce chunk size for inserting dervation inputs
...
As this query can take some time.
2022-09-14 09:48:59 +01:00
77c4e1cb63
Reduce the chunk size for querying related derivation ids
...
And include the chunk size in the log message.
2022-09-13 21:00:04 +01:00
ad39492cf0
Support per repository configuration for fetching substitutes
...
This is so that data.qa.guix.gnu.org can be configured only to query the
branches from the main repository.
2022-09-09 12:39:38 +01:00
45e25c2436
Make caching exceptions when querying for substitutes more reliable
2022-09-09 12:33:15 +01:00
076331325a
Log more information about heap size when loading derivation info
...
To better understand the memory usage when this is happening.
2022-09-05 14:23:10 +01:00
b3d59c650a
Use much smaller chunks when trying to run the derivation linter
...
Since larger chunks still ran in to inferior memory usage problems.
2022-09-05 14:22:38 +01:00
aa8c9dbffa
Compute lint warnings for packages in chunks
...
In an attempt to reduce the peak memory usage, and avoid running in to the:
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
issue.
2022-09-05 08:40:27 +01:00
1043b51bff
Include lint warnings in the JSON /compare response
2022-08-22 17:42:37 +01:00
ee73d2cc98
Return 503 for the /healthcheck if the service is starting up
...
As this is a little clearer.
2022-07-15 09:24:46 +01:00
1962b1f61c
Fix missing NULL variable
2022-07-09 17:14:55 +01:00
39487cd7e6
Improve deleting derivations
...
Drop the batch size to get rid of warnings about memory usage and improve the
logging by adding duration information.
2022-07-08 20:55:58 +01:00
788571f53f
Set builds derivation output details set id if information available
...
The build event information can now contain the derivation outputs, as well as
the name of the derivation. This allows the Guix Data Service to join builds
up with derivations, even if it doesn't know about the derivation being built.
2022-07-08 13:48:08 +01:00
6da5e8e67b
Sort derivation output details ids
...
To ensure that direct array comparison can be used in the query.
2022-07-08 13:47:52 +01:00
db37d9f6a8
Split out inserting derivation output details sets
...
So that this can be used when inserting builds.
2022-07-08 13:47:52 +01:00
811256a920
Split out inserting into derivation_output_details
...
So that this can be done when inserting builds.
2022-07-08 13:47:52 +01:00
