cbaines/guix-data-service
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Rework how data is inserted

Browse source 
This is the big change needed to allow for parallel revision
processing. Previously, a lock was used to prevent this since the parallel
transactions could deadlock if each inserted data that the other then went to
insert.

By defining the order in which inserts happen, both in terms of the order of
tables, and the order of rows within the table, this change should guarantee
that there won't be deadlocks.

I'm also hoping this change will address whatever issue was causing some
derivation data to be missing from the database.
This commit is contained in:
Christopher Baines 2025-12-28 19:01:39 +00:00
parent 72fe3b4e47
commit 66793a5568
25 changed files with 3371 additions and 1825 deletions
Download patch file Download diff file Expand all files Collapse all files

26
tests/jobs-load-new-guix-revision.scm
View file

@ -3,11 +3,13 @@
#:use-module (ice-9 match)
#:use-module (squee)
#:use-module (fibers)
#:use-module (knots)
#:use-module (guix utils)
#:use-module (guix store)
#:use-module (guix tests)
#:use-module (guix-data-service database)
#:use-module (guix-data-service model git-repository)
#:use-module (guix-data-service model guix-revision)
#:use-module (guix-data-service jobs load-new-guix-revision))
(test-begin "jobs-load-new-guix-revision")
@ -65,13 +67,16 @@
((guix-data-service jobs load-new-guix-revision)
extract-information-from
(lambda _
#t))
'()))
(mock
((guix-data-service model channel-instance)
insert-channel-instances
(lambda (conn guix-revision-id derivations-by-system)
#t))
((guix-data-service jobs load-new-guix-revision)
load-channel-instances
(lambda (call-with-utility-thread
read-derivations/serialised
git-repository-id commit
channel-derivations-by-system)
(insert-guix-revision conn git-repository-id commit)))
(mock
((guix channels)
@ -81,7 +86,7 @@
(mock
((guix-data-service jobs load-new-guix-revision)
derivation-file-names->derivation-ids
insert-derivations-with-table-managers
(lambda _
#(1)))
@ -103,8 +108,13 @@
((id)
(run-fibers
(lambda ()
(process-load-new-guix-revision-job
id #:parallelism 1))
(with-exception-handler
(lambda (exn)
(print-backtrace-and-exception/knots exn)
(raise-exception exn))
(lambda ()
(process-load-new-guix-revision-job
id #:parallelism 1))))
#:hz 0
#:parallelism 1
#:drain? #t))))))))))))))