Commit graph

17 commits

Author SHA1 Message Date
2073e446b7 Tweak the job script
Don't use with-fluids (I forget why), and enable core dumps.
2025-07-09 12:50:24 +01:00
7c0779519b Allow specifying a limit to inferior memory usage
To help manage the inferiors that use gigabytes of memory while computing
derivations.
2025-06-28 09:29:10 +02:00
0dd14c0a67 Use drain? #t for fibers when loading revisions
To check that there's no left over fibers.
2025-06-28 09:29:10 +02:00
931b7bc593 Add a slightly crude method to ignore systems and targets
While processing a revision. It would be good to also record what systems and
targets are in the platforms so it's clear what data is missing, but that can
be added later.
2025-02-03 22:59:34 +00:00
31bd2156f7 Support setting environment variables in the inferior
When processing jobs, this is mostly to allow setting GUIX_DOWNLOAD_METHODS.
2024-06-24 23:02:14 +01:00
a900a1c2ec Remove drain? #t from process job
As it now uses more fibers.
2024-01-18 22:41:02 +00:00
c1d2f3a1b7 Add meaningful parallelism to processing jobs
Make parallel use of inferiors when computing channel instance derivations,
and when extracting information about a revision. This should allow for some
horizontal scalability, reducing the impact of additional systems for which
derivations need computing.

This commit also fixes an apparent issue with package replacements, as
previously the wrong id was used, and this hid some issues around
deduplication.
2024-01-18 15:34:40 +00:00
a3ec1f326d Set %file-port-name-canonicalization when processing jobs
Just in case this helps with performance.
2023-12-04 11:06:27 +00:00
c3cb04cb80 Use fibers when processing new revisions
Just have one fiber at the moment, but this will enable using fibers for
parallelism in the future.

Fibers seemed to cause problems with the logging setup, which was a bit odd in
the first place. So move logging to the parent process which is better anyway.
2023-11-05 13:46:20 +00:00
3ba8418656 Allow skipping processing system tests
Generating system test derivations are difficult, since you generally need to
do potentially expensive builds for the system you're generating the system
tests for. You might not want to disable grafts for instance because you might
be trying to test whatever the test is testing in the context of grafts being
enabled.

I'm looking at skipping the system tests on data.guix.gnu.org, because they're
not used and quite expensive to compute.
2023-02-08 14:56:48 +00:00
39b5df04eb Remove development code from the process job script 2020-09-28 08:29:20 +01:00
033858410b Add a JSON page for repository branches 2020-09-27 16:32:56 +01:00
fb180e1ebd Replace debug-set! with setenv COLUMNS
As that actually seems to work.
2020-09-26 16:42:18 +01:00
53341c70fc Change the locale codeset representation
From the normalized one, to the one actually contained within glibc. Recent
versions of glibc also contain symlinks linking the normalized codeset to the
locales with the .UTF-8 ending, but older ones do not.

Maybe handling codeset normalisation for queries would be good, but the locale
values ending in .UTF-8 are more compatible and allow the code to be
simplified. For querying, maybe there should be a locales table which handles
different representations.
2020-09-26 11:45:57 +01:00
e38db9eed9 Set the locale at the start of the process jobs script
This might help with the odd [1] errors regarding PostgreSQL queries.

1: invalid byte sequence for encoding "UTF8":
2020-09-20 11:11:03 +01:00
a0e098a6ce Increase the stack trace width when processing jobs
As this might result in more useful error messages.
2020-09-20 10:59:22 +01:00
09d927cb99 Add a process-job command
This allows easily processing an individual job by id. This may be useful to
use manually, but also when processing jobs in parallel, as forking doesn't
work well with the libpq library used by squee.
2019-07-12 19:57:24 +01:00