e51d2f8932
Remove some time logging
...
As this is a bit noisy.
2024-01-27 18:39:41 +00:00
15b6dad5a5
Have delete-duplicates/sort! take a equality procedure
...
And change the default, as eq? doesn't always work.
2024-01-18 14:41:32 +00:00
4f1ae74d2f
Handle derivations with no sources
2023-11-05 18:49:23 +00:00
03327c0cc3
Include output information in the package page response
...
As this will be useful for QA to say whether the package builds reproducibly
or not.
2023-11-05 13:46:20 +00:00
f5acc60288
Make some sweeping changes to loading new revisions
...
Move in the direction of being able to run multiple inferior REPLs, and use
some vectors rather than lists in places (maybe this is more efficient).
2023-11-02 12:16:17 +00:00
54c7a1a880
Fix ignoring canceled builds
...
The previous changes only affected searching for package derivations, and they
also didn't work.
2023-05-18 12:31:58 +01:00
4208b5f148
Ignore canceled builds when querying package derivations
...
This will help when using this to submit builds, since you won't end up
ignoring derivations with canceled builds.
2023-05-18 11:25:14 +01:00
1a0eaeb672
Improve performance of select-fixed-output-package-derivations-in-revision
2023-03-11 18:19:19 +00:00
6ada1cb845
Guard against divide by 0 in update-derivation-outputs-statistics
2022-11-28 13:17:20 +00:00
1a0c5599eb
Do derivation inputs and outputs housekeeping at the end of each job
...
This should help with query performance, as the recursive queries using
derivation_inputs and derivation_outputs are particularly sensitive to the
n_distinct values for these tables.
2022-11-28 11:36:12 +00:00
78a5abee21
Improve chunking when inserting derivation inputs
...
Chunk the values inserted in the query, rather than the derivations involved,
as this is more consistent.
2022-09-17 08:53:23 +02:00
7050ea749f
Reduce some chunk sizes
2022-09-17 00:40:51 +02:00
8ef896b103
Further reduce some chunk sizes
2022-09-15 16:25:31 +02:00
f41bfcf8b6
Reduce some chunk sizes
...
As these queries are still slow enough to be logged.
2022-09-14 15:42:00 +01:00
12af30c039
Reduce chunk size for inserting dervation inputs
...
As this query can take some time.
2022-09-14 09:48:59 +01:00
77c4e1cb63
Reduce the chunk size for querying related derivation ids
...
And include the chunk size in the log message.
2022-09-13 21:00:04 +01:00
6da5e8e67b
Sort derivation output details ids
...
To ensure that direct array comparison can be used in the query.
2022-07-08 13:47:52 +01:00
db37d9f6a8
Split out inserting derivation output details sets
...
So that this can be used when inserting builds.
2022-07-08 13:47:52 +01:00
811256a920
Split out inserting into derivation_output_details
...
So that this can be done when inserting builds.
2022-07-08 13:47:52 +01:00
6d403cbc8d
Allow filtering package derivations based on build server builds
...
This means you can query for derivations where builds exist or don't exist on
a given build server.
I think this will come in useful when submitting builds from a Guix Data
Service instance.
2022-05-23 22:39:32 +01:00
df4e0a7a61
Add to the hardcoded list of valid targets
...
Since the hardcoded list in the load-new-guix-revision code has been updated.
2022-03-11 11:50:10 +00:00
f86657915e
Try to further speed up inserting missing derivation source files
...
Switch from using a recursive query to doing a breath first search through the
graph of derivations, as I think PostgreSQL wasn't doing a great job of
planning the recursive queries (it would overestimate the rows involved, and
prefer sequential scans for the derivation_outputs table).
2022-03-02 18:00:36 +00:00
c5b504e94a
Speed up the finding of missing sources
...
Use larger batches and more efficient duplicate deletion.
2022-03-01 20:57:26 +00:00
f1418c4e88
Support querying package derivation outputs without the nars
...
Since this speeds up the response if you don't need the nar information.
2022-01-31 20:24:27 +00:00
a7c9daab6a
Process derivations in chunks
...
Which should reduce the peak memory usage.
2022-01-14 15:25:53 +00:00
5ae8b796a7
Rename chunk-map! to chunk-for-each!
...
As that better reflects what it does.
2022-01-14 15:25:13 +00:00
21cb33a859
Re-write insert-derivation-inputs in a more memory efficient manor
...
Previously it would compute a long list of strings, potentially more than
100,000 elements long, then split this string up and insert it in chunks. Only
then could memory be freed.
This new approach builds the strings in batches for the insertion query, then
moves on to the next batch. This should mean that more memory can be freed and
reused along the way.
2022-01-12 18:18:15 +00:00
ba9bcbf735
Use a bigger start size for the hash table
...
This might help when there's lots of derivations to insert.
2021-10-03 15:28:40 +01:00
b28d338de7
Insert derivations in chunks
...
To avoid making a very large query when inserting lots of derivations.
2021-10-03 14:54:43 +01:00
af0a06d147
Log the time to read missing derivations from the store
2021-10-03 12:59:26 +01:00
3627d36d77
Select existing derivations in chunks
...
To avoid one massive query.
2021-10-03 12:59:02 +01:00
857b4e32d5
Insert derivation inputs in chunks
...
To avoid one massive query.
2021-10-03 12:56:23 +01:00
211da6868f
Handle the case where there are no missing file names
...
In update-derivation-ids-hash-table!.
2021-09-25 00:09:08 +01:00
3081887b90
Optimise inserting derivation inputs
...
Rather than querying for the output ids one by one and then running an insert
query for each derivation, perform the task with a single insert query.
2021-09-24 18:22:28 +01:00
abff41f9ae
Neaten up formatting in select-derivation-output-id
2021-09-24 17:26:48 +01:00
947cabb685
Replace derivation-file-names->vhash
...
Rather than creating vhashes, just update the hash table that is used as a
cache, and query that. This should speed things up and reduce memory usage
when loading derivations.
2021-09-24 17:14:40 +01:00
261552bd5e
Speed up inserting missing derivation sources
...
Split the recursive part of the query from the non-recursive part, since
PostgreSQL doesn't do a great job of estimating the number of rows which will
come back from the recursive part, and thus generates a bad plan.
2021-09-24 12:55:45 +01:00
8b34126d22
Fix the select-source-files-missing-nars query
...
Previously, the VALUES list wasn't being generated properly.
2021-09-16 17:01:34 +01:00
f2b9663cf2
Make sure to add missing source file nars when inserting derivations
...
Source file nars weren't always recorded, so this will help backfill that
data (which enables providing substitutes for those derivations).
2021-09-14 21:19:48 +01:00
673d762e68
Remove redundant derivation-sources call
2021-06-09 16:38:33 +01:00
6387f1bc67
Fetch the list of system values from the database
...
This removes the need to hardcode some values in the code.
2021-04-23 11:20:36 +01:00
b430c632b7
Create a table for systems
...
And use it for the systems in the derivations and package derivations tables.
The primary motivation here is to allow quickly working out what systems the
database contains, and having a small table with just the right data seems a
good way to do that.
2021-04-23 11:14:51 +01:00
9dde93422c
Add powerpc64le-linux as a supported system
...
This should not be necessary, but currently the database isn't used to find
out the valid system values.
2021-02-12 08:47:09 +00:00
f868591d1c
Show finished builds on the fixed output package derivations page
...
Rather than scheduled builds as well, as that's not useful information.
2021-02-02 19:37:35 +00:00
54c5fbe9fb
Fix a Squee null handling issue with latest fixed output derivations
2021-01-14 22:09:32 +00:00
7b66253639
Add missing build server build ids to various queries
2021-01-03 11:05:29 +00:00
e85131a004
Fix squee compatability issues for serving derivations
2021-01-02 10:55:25 +00:00
64a4058cce
Start to add compatibility with squee returning #f for null values
...
While maintaining compatibility for older versions of squee.
2021-01-02 10:06:27 +00:00
7df6f92036
Improve select-fixed-output-package-derivations-in-revision
...
Refactor the query to make it more performant.
2020-12-29 18:14:20 +00:00
844bd32f22
Add select-fixed-output-package-derivations-in-revision
2020-12-26 13:39:45 +00:00
