Commit graph

34 commits

Author SHA1 Message Date
05c3ae5b87 Use more threads for the server
Just in case this avoids writing responses blocking new requests coming in.
2024-08-14 21:42:11 +01:00
22eb8d44ca Move the resource pools to a different thread
As the server thread can block for a while when writing big responses.
2024-08-14 21:13:55 +01:00
b2dcccb264 Name the resource pools 2024-08-14 20:12:19 +01:00
d8ade0ea11 Fix the pool metrics 2024-07-29 11:24:24 +01:00
e8bd27fdb8 Try using 2 threads for the server 2024-07-26 21:42:21 +01:00
e81c6377bf Re-work the fibers scheduling
Use a single thread for receiving and responding to requests, and delegate the
processing of the requests to a separate set of threads.

I'm hoping this will avoid the processing of requests affecting accepting new
connections, or the sending of responses.
2024-07-16 23:03:15 +01:00
d8e9de4ed6 Use a bigger buffer for requests/responses
As I'm hoping this will increase the data throughput for responses.
2024-07-16 20:37:07 +01:00
6201099003 Instrument requests 2024-04-26 09:46:10 +01:00
c18589249f Instrument the resource pool checkout timeouts 2024-04-15 13:54:25 +01:00
65f92ca701 Guard against logging failing
As I've seen this happening where there are errors writing to FD 1.
2024-04-02 12:16:15 +01:00
6808532338 Close unused PostgreSQL connections
As they take up memory, so free some by closing them.
2024-04-01 22:20:46 +01:00
b5f59189e1 Move backfilling in to the server module and use the connection pool
To avoid using the old PostgreSQL connection per thread code.
2024-04-01 21:51:29 +01:00
26a751570c Copy the port monitoring fiber from the build coordinator
As the data service has the same issue where it stops listening on the port.
2024-02-13 09:59:39 +00:00
241a704db1 Instrument handling build events 2023-11-24 16:11:42 +00:00
de211ef5ea Set the names of more threads
To help with debugging.
2023-11-16 11:30:23 +00:00
f9bb60ab4a Drop the resource pool timeout
Better to timeout early.
2023-07-14 10:35:44 +01:00
22ed81fbcc Reduce the number of threads used by fibers
As one thread per core is probably unnecessary.
2023-07-10 18:56:31 +01:00
75ef4cffd3 Expose resource pool stats 2023-07-10 18:56:31 +01:00
7251c7d653 Stop using a pool of threads for database operations
Now that squee cooperates with suspendable ports, this is unnecessary. Use a
connection pool to still support running queries in parallel using multiple
connections.
2023-07-10 18:56:31 +01:00
d29ef3ed9b Enable database connection instrumentation in the server 2023-01-01 12:43:19 +00:00
926cb2a5e1 Pull the metrics registry out of the controller
This will allow for instrumenting low level database functionality, before
anything starts using the database.
2023-01-01 12:27:34 +00:00
8e23d38660 Handle migrations and server startup better
The server part of the guix-data-service doesn't work great as a guix service,
since it often fails to start if the migrations take any time at all.

To address this, start the server before running the migrations, and serve the
pages that work without the database, plus a general 503 response. Once the
migrations have completed, switch to the normal behaviour.
2022-06-17 13:13:21 +01:00
18b6dd9e6d Stop opening a PostgreSQL connection per request
This was good in that it avoided having to deal with long running connections,
but it probably takes some time to open the connection, and these changes are
a step towards offloading the PostgreSQL queries to other threads, so they
don't block the threads for fibers.
2020-10-03 09:22:29 +01:00
a0263a0eae Set a statement timeout of 60 seconds for web requests
This will help stop queries running for an unnecessarily long time, longer
than NGinx will wait for example.
2020-04-24 09:00:20 +01:00
865e856e8f Provide more information if the server can't bind the port
As this could be a common problem.
2020-03-14 13:13:48 +00:00
5663235048 Rework the builds and build_status tables as well as related code
Allow for build status information to be submitted by POST request. This
required some changes to the builds and build_status tables, as for example,
the Cuirass build id may not be available, and the derivation may not be know
yet, so just record the derivation file name.
2019-11-24 20:18:08 +00:00
3a073d402d Display the request in one go
Otherwise, concurrent requests can cause the output to be interleaved.
2019-06-13 19:07:07 +01:00
5323f24133 Add --port and --host as options to the guix-data-service 2019-06-13 19:06:53 +01:00
aad2c9d9e8 Extract the database connection handling from the server
Previously, one of the first things that happened when responding to a request
was a database connection was made, even when serving the CSS. This is
unnecessary, so move the database connection handling in to the controller.

Also, to allow for separating it out from the assets, separate the assets out
from the parts of the controller that require a database connection.
2019-06-06 20:43:54 +01:00
658a1a20b2 Improve the content negotiation handling in general
Previously, the routing layer handled the content negotiation, and the Accept
header was ignored. Now, the extension if one is provided in the URL is still
used, and more widely than before, but the Accept header is also taken in to
account.

This all now happens before the routing decisions are made, so the routing is
now pretty much extension independant (with the exception of the
/gnu/store/... routes).
2019-05-11 22:56:25 +01:00
2e45bb0696 Extract out some database functionality to another module 2019-04-14 10:04:51 +01:00
800c850276
Switch dynamic-wind for with-throw-handler
Not sure why, but dynamic-wind was causing issues by closing the database
connection too early, but with-throw-handler seems to work OK.
2019-03-24 17:35:09 +00:00
0a49c0a84a
Actually close database connections
Previously, the connections were not closed, so eventually PostgreSQL
would run out. Using a pool of connections would be better, but as a
short term solution, just close the connection after each request.
2019-02-08 11:19:12 +00:00
5a9262b38d
Initial commit
This is a service designed to provide information about Guix. At the
moment, this initial prototype gathers up information about packages,
the associated metadata and derivations.

The initial primary use case is to compare two different revisions of
Guix, detecting which packages are new, no longer present, updated or
otherwise different.

It's based on the Mumi project.

[1]: https://git.elephly.net/software/mumi.git
2019-02-07 22:26:57 +00:00