Commit graph

105 commits

Author SHA1 Message Date
e38db9eed9 Set the locale at the start of the process jobs script
This might help with the odd [1] errors regarding PostgreSQL queries.

1: invalid byte sequence for encoding "UTF8":
2020-09-20 11:11:03 +01:00
a0e098a6ce Increase the stack trace width when processing jobs
As this might result in more useful error messages.
2020-09-20 10:59:22 +01:00
b6754c8a4c Add a lookup_builds field to the build_servers table
This is to allow for build servers where only the substitutes should be
queried, and it shouldn't be assumed that they're running Cuirass.
2020-05-24 17:02:53 +01:00
9c72fc23dc Move around --no-tablespaces
Turns out, at the moment, this is ineffective when combined with the archive
formats, like the custom format in use. Therefore, move it to the pg_restore
command, where hopefully it'll work.
2020-05-16 08:42:00 +01:00
796c129a36 Don't include tablespace assignments in the backup dump
This is a comprimise, as this won't help restoring the backup in situations
you want tablespaces, but I'm currently viewing tablespaces as a deployment
concern, so maybe the right thing to do is exclude them. This approach will at
least keep the same behaviour in terms of restoring the backups locally.

This will fix the small dump creation process on data.guix.gnu.org, which is
currently broken because of the tablespace assignments when trying to restore
the backups.
2020-05-14 20:49:46 +01:00
6baef6ae25 Split out querying of build servers and substitute servers
These are related things, but somewhat separate. This change should make it
easier to deal with changes regarding querying build servers, and querying
substitute servers.
2020-05-03 13:23:43 +01:00
a0263a0eae Set a statement timeout of 60 seconds for web requests
This will help stop queries running for an unnecessarily long time, longer
than NGinx will wait for example.
2020-04-24 09:00:20 +01:00
5081a64c1f Rebuild the package derivation ranges table for the small backup
This is better than just deleting the entries that don't match up with the
remaining revisions, but also not very useful for local development (due to
the lack of data).
2020-03-31 20:46:18 +01:00
d1c243f7fd Give the temporary database more working memory
In the hope that this makes the script faster.
2020-03-26 20:21:47 +00:00
3017765f0c Use EXPLAIN ANALYZE for the creation of tmp_derivations
In the create-small-backup script, as this is quite a slow part, it's useful
to get more information.
2020-03-26 20:21:14 +00:00
9a79a5d747 Handle a couple more tables in create-small-backup
derivation_output_details_sets, and derivations_by_output_details_set. This
required moving around some of the code.
2020-03-26 20:20:29 +00:00
d0eff9da5d Use the --no-comments option to pg_dump
Hopefully this will help with the pg_restore in the create-small-backup
script:

  pg_restore: [archiver (db)] Error while PROCESSING TOC:
  pg_restore: [archiver (db)] Error from TOC entry 2875; 0 0 COMMENT EXTENSION plpgsql
  pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
      Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
2020-03-25 20:47:53 +00:00
8af7130239 Handle channel instances in create-small-backup
Otherwise this table is empty.
2020-03-25 18:27:01 +00:00
b99854924a Handle system test derivations in create-small-backup
Otherwise this table is empty.
2020-03-25 18:26:33 +00:00
ca0d3ee754 Stop using package_versions_by_guix_revision_range
It's been replaced by the package_derivations_by_guix_revision_range table.
2020-03-24 20:44:57 +00:00
cf4082dbeb Avoid failures related to renice and ionice
These parts of the backup scripts are optional, so don't fail if they don't
work.
2020-03-20 20:40:33 +00:00
ded4df6632 Move and improve the "starting the server" message
Move it after the output relating to narinfo signing, and include the host.
2020-03-14 13:14:29 +00:00
a03e1601de Improve handling of errors
Adjust the previously unused error page code, and start to use it. Only show
the error if configured to do so, to avoid leaking secret information.
2020-03-14 12:46:02 +00:00
baeae56de4 Don't use TRUNCATE CASCADE in the create small backup script
As it makes it clearer what tables will be truncated.
2020-03-13 18:38:42 +00:00
6ce96ad55b Trip the derivation output details table in the small data dump 2020-03-13 18:38:29 +00:00
b64e6b19c2 Trim derivation source file tables in the small data dump 2020-03-13 18:37:46 +00:00
77caafb019 Add scripts for generating database dumps 2020-03-02 21:44:29 +00:00
65f2f21d3a Support customising the latest branch revision max processes
This makes it possible to set a higher or lower value depending on what you
want.
2020-02-28 20:58:21 +00:00
9c7310f8e3 Make it possible to query builds servers for specific outputs 2020-02-16 09:54:27 +00:00
33749786e4 Add verbose output to the query-build-servers script 2020-02-15 19:54:42 +00:00
381acf312e Warn if process-branch-updated-mbox won't match any emails 2020-02-01 14:03:26 +01:00
7342280ead Guard against failures when loading the narinfo signing public key 2019-12-26 10:49:14 +00:00
54b3d8d93a Fix the option alist element names for the narinfo signing key 2019-12-26 10:47:29 +00:00
66e886a6b4 Serve narinfo files for derivations 2019-12-26 09:34:43 +00:00
96305ec406 Fix the query-build-servers script when no commits are provided 2019-12-23 19:27:23 +00:00
c516154816 Fix the default max-processes for the process-jobs script 2019-12-22 23:26:45 +00:00
c229ace8a5 Allow customising the maximum number of processes 2019-12-16 19:57:56 +00:00
c90f411e5e Add options to the query-build-servers script
So you can select to query specific build servers.
2019-12-12 20:07:22 +00:00
7cc5c02cdd Query build servers for narinfo files
As well as making it possible to only query build servers for builds relating
to a specific revision.
2019-11-30 10:59:16 +00:00
5663235048 Rework the builds and build_status tables as well as related code
Allow for build status information to be submitted by POST request. This
required some changes to the builds and build_status tables, as for example,
the Cuirass build id may not be available, and the derivation may not be know
yet, so just record the derivation file name.
2019-11-24 20:18:08 +00:00
0ffd8caeeb Add a script to help manage build servers
This computes and displays the tokens needed to send build events to the Guix
Data Service.
2019-11-23 11:50:51 +00:00
977609c66e Rewrite localhost for sqitch
As this means that like psql, the unix socket will be used.
2019-10-01 19:03:34 +01:00
ea253c4515 Add a new script to process emails in an mbox file
This can be used with the mbox files for the guix-commits mailing list to add
older emails in to the database.
2019-09-28 23:55:11 +01:00
759b75257e Switch to processing emails as bytevectors
This is better, as different parts of the email might be encoded differently,
and guile-email will take care of this if handed a bytevector.
2019-09-26 18:08:12 +01:00
d35cc85d52 Improve error handling for processing emails 2019-08-31 12:06:36 +01:00
09d927cb99 Add a process-job command
This allows easily processing an individual job by id. This may be useful to
use manually, but also when processing jobs in parallel, as forking doesn't
work well with the libpq library used by squee.
2019-07-12 19:57:24 +01:00
743cec7cb6 Associate a name with database connections
This helps when working out which connection to the database is doing what.
2019-07-12 19:49:26 +01:00
3df0b43146 Use line buffering for the input and output ports
As these are used for logging, which is done on a line by line basis. Remove
the now redundant calls to (force-output).
2019-06-17 18:09:10 +01:00
5323f24133 Add --port and --host as options to the guix-data-service 2019-06-13 19:06:53 +01:00
bd292d6f57 Add an option to create a pid-file
This'll help the a service manager (like the shepherd) know when the service
is ready, which at the moment, means the database migrations have happened.
2019-06-11 07:40:26 +01:00
cf5dace846 Change to the sqitch directory to run deploy
As this means it can find the relevant files.
2019-06-11 07:22:17 +01:00
93fdefc7b1 Pass the --db-client option to sqitch deploy
So that it can always find psql.
2019-06-11 07:15:01 +01:00
d7586e88fa Pass --plan-file to the sqitch deploy command 2019-06-08 10:28:14 +01:00
42c608e004 Add a --update-database option
This runs Sqitch on startup, which should make managing the database easier,
as you just have to restart the service with this option, and the database
should be updated if necessary.
2019-06-07 07:33:34 +01:00
7a34570ef8 Add missing (guix-data-service database) import 2019-04-17 08:28:36 +01:00