cbaines/guix-data-service
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
guix-data-service/sqitch/sqitch.plan

51 lines
6.4 KiB
Text
Raw Normal View History

Initialise Sqitch It's a tool for managing the database schema.
2019-04-13 12:22:14 +01:00
%syntax-version=1.0.0
%project=guix-data-service
%uri=https://git.cbaines.net/guix/data-service/
Add some initial Sqitch migrations These are based on the state of the current manually managed database.
2019-04-14 10:06:13 +01:00
appschema 2019-04-13T11:43:59Z Christopher Baines <mail@cbaines.net> # Add schema for the Guix Data Service
buildstatus_enum [appschema] 2019-04-13T11:56:37Z Christopher Baines <mail@cbaines.net> # Creates the buildstatus enum
initial_import 2019-04-13T13:06:28Z Christopher Baines <mail@cbaines.net> # Import the manually managed database schema
Switch to storing Git repositories in a table Rather than just storing the URL in the guix_revisions and load_new_guix_revision_jobs tables. This will help when storing more information like tags and branches in the future.
2019-05-05 13:35:48 +01:00
git_repositories 2019-05-04T19:03:38Z Christopher Baines <mail@cbaines.net> # Add a git_repositories table
Start to handle information about Git branches Add some new pages /branches and /branch/... as well as a new git_branches table. Also extend the email processing to enter the branch information in to the database.
2019-05-05 20:06:28 +01:00
git_branches 2019-05-05T14:53:12Z Christopher Baines <mail@cbaines.net> # Add a git_branches table
Remove the sha1_hash from the package_metadata table I'm thinking about adding more fields to this table, and the sha1_hash values will make this tricker. Therefore, remove the value, and adjust the existing code to cope. This commit also adds a new test which coveres some of the changed functionality.
2019-05-12 17:01:51 +01:00
remove_package_metadata_sha1_hash 2019-05-12T10:37:40Z Christopher Baines <mail@cbaines.net> # Remove the sha1_hash field from package_metadata
Store and display the location of packages Store the location a package can be found at, and display this on the package page. If available, link off to the git repository containing the package.
2019-05-13 21:02:53 +01:00
add_location_information 2019-05-12T20:27:48Z Christopher Baines <mail@cbaines.net> # Add locations table and location to package_metadata
add_git_repositories_cgit_url_base 2019-05-13T18:45:14Z Christopher Baines <mail@cbaines.net> # Add cgit_url_base to git_repositories
Store license information for packages And display this on the package page. This uses a couple of new tables, and an additional field in the package_metadata table. Currently, the order of the licenses in the package definition isn't stored, as I'm not sure the order in the list is significant.
2019-05-15 08:05:14 +01:00
license_support 2019-05-13T20:37:40Z Christopher Baines <mail@cbaines.net> # Add support for storing license information
Record job success without deleting the job record Previously, the records for jobs would be deleted. It's useful to know when jobs were inserted in to the database, as well as when they succeeded (if they have). This change also makes it possible to keep track of jobs that have failed, as they won't be deleted.
2019-06-02 16:20:51 +01:00
dates_to_load_new_guix_revision_jobs 2019-06-02T07:39:49Z Christopher Baines <mail@cbaines.net> # Add dates to the load_new_guix_revision_jobs table
Add more detailed new revision job handling Create a new events table for the new guix revision jobs, and update this when processing a job starts, as well as finished with success or failure. Additionally, remove the dependnency on open-inferior/container, as this functionality isn't merged in to Guix master yet.
2019-06-02 22:00:29 +01:00
load_new_guix_revision_job_events 2019-06-02T15:44:41Z Christopher Baines <mail@cbaines.net> # Add new table for guix_revision_job_events
Record the output from loading new revisions to the database So that it can easily be shown through the web interface. There's two tables being used. One which temporarily stores the output as it's output while the job is running, and other which stores the whole log once the job has finished.
2019-06-22 01:13:36 +02:00
load_new_guix_revision_job_logs 2019-06-21T14:33:09Z chris <chris@phact> # Add load_new_guix_revision_job_logs
Tweak how logs are stored Previously, the query for the jobs page was really slow, as it checked the load_new_guix_revision_job_log_parts table for each job, doing a sequential scan through the potentially large table. Adding an index didn't seem to help, as the query planner would belive the query could return loads of rows, where actually, all that needed checking is whether a single row existed with a given job_id. To avoid adding the index to the load_new_guix_revision_job_log_parts table, and fighting with the query planner, this commit changes the load_new_guix_revision_job_logs table to include a blank entry for jobs which are currently being processed. This is inserted at the start of the job, and then updated at the end to combine and replace all the parts. This all means that the jobs page should render quickly now.
2019-07-07 21:23:18 +01:00
change_load_new_guix_revision_job_logs_contents_to_be_nullable 2019-07-07T20:10:54Z Christopher Baines <mail@cbaines.net> # Change the contents field in the load_new_guix_revision_job_logs table\nto be nullable.\n\nwith '#' will # be ignored, and an empty message aborts the add. #\nChange to add: # #\nchange_load_new_guix_revision_job_logs_contents_to_be_nullable #\nsqitch/deploy/change_load_new_guix_revision_job_logs_contents_to_be_nullable.sql\nsqitch/revert/change_load_new_guix_revision_job_logs_contents_to_be_nullable.sql\nsqitch/verify/change_load_new_guix_revision_job_logs_contents_to_be_nullable.sql
Fix some duplicated values in tables The licenses table, along with the package_metadata table had duplicate values. This could happen as the unique constraints on those tables didn't properly account for the nullable fields. The duplicates in those tables also affected the license_sets, packages, package_derivations tables in a similar way. Finally, the guix_revision_package_derivations table was also affected. This commit adds a migration to fix the data, as well as the constraints. THe code to populate the licenses and package_metadata tables is also updated.
2019-08-04 09:39:40 +01:00
fix_duplicated_licenses 2019-07-30T05:48:17Z Christopher Baines <mail@cbaines.net> # Fix duplicated licenses, and add constraints
Change the git_branches table primary key To allow for having branches in multiple git repositories.
2019-08-05 20:02:06 +01:00
change_git_branches_primary_key 2019-08-05T18:57:41Z Christopher Baines <mail@cbaines.net> # Change the git_branches primary key to include the git_repository_id,\nas this will allow having the same branch in different repositories.
Avoid erroring when processing emails again These changes allow processing emails again, and just creating job and branch entries where data is missing.
2019-08-05 20:59:48 +01:00
remove_duplicate_load_new_guix_revision_jobs 2019-08-05T19:06:36Z Christopher Baines <mail@cbaines.net> # Remove duplicate load_new_guix_revision_jobs
Store lint warnings in the database This commit adds the relevant tables and code to store lint warnings in the database. Currently, only lint checkers which don't require access to the network will be run, as this allows the processing to happen without network access. Also, this functionality won't work in older versions of Guix which don't expose the lint warnings in a compatible way.
2019-08-31 12:11:58 +01:00
lint_warnings 2019-08-18T17:10:12Z Christopher Baines <mail@cbaines.net> # Store lint warnings
Add a new table guix_revision_lint_checkers To associate a set of lint checkers with a specific revision. While there is the association through the lint warnings, that only works for checkers where there are lint warnings for that revision. Therefore, to allow finding all the checkers for a specific revision, also associate them directly with the revision.
2019-09-01 13:35:23 +01:00
guix_revision_lint_checkers 2019-09-01T12:17:38Z chris <chris@phact> # Associate lint_checkers to guix_revisions
Add a new table to store package versions by revision ranges This isn't new information, but derived from information already in the database. It's collected here to make querying faster. The table is updated when each new revision is entered.
2019-09-26 23:45:07 +01:00
package_versions_by_guix_revision_range 2019-09-26T20:23:15Z Christopher Baines <mail@cbaines.net> # Add package_versions_by_guix_revision_range table
Add migration to create an index on load_new_guix_revision_job_events To speed up the rendering of the index page.
2019-09-29 12:04:57 +01:00
index_on_load_new_guix_revision_job_events_job_id 2019-09-29T10:39:04Z Christopher Baines <mail@cbaines.net> # Add index on load_new_guix_revision_job_events.job_id
Fix the 'NULL' values in git_branches for the commits The git_branches table had 'NULL' values for some commits where the branch was deleted, importantly this was the string 'NULL', not an actual NULL value. This commit fixes that, migrating the existing values to be '', and changing the relevant code. The primary key is also extended to include the datetime field, as this is important to allow a branch to be deleted twice.
2019-09-29 17:10:04 +01:00
fix_null_values_in_git_branches 2019-09-29T11:06:12Z Christopher Baines <mail@cbaines.net> # Fix the NULL values in git_branches
Make it easier to retry jobs Add a new event 'retry', and run jobs where the number of retry events is greater or equal to the number of failure events. Also add an index to the git_branches table to make the finding jobs query a bit faster.
2019-10-02 21:54:11 +01:00
add_retry_value_to_job_event_enum 2019-10-02T19:13:52Z Christopher Baines <mail@cbaines.net> # Add retry value to job_event enum
Remove duplicates from the guix_revisions table
2019-10-05 09:14:40 +01:00
remove_guix_revision_duplicates 2019-10-05T08:00:14Z Christopher Baines <mail@cbaines.net> # Remove duplicates in the guix_revisions table
Add a new table to describe the history of derivations There's already the package_versions_by_guix_revision_range table, but I think it would be also useful to be able to see how derivations change over time.
2019-11-09 19:59:55 +00:00
package_derivations_by_guix_revision_range 2019-11-09T19:09:48Z Christopher Baines <mail@cbaines.net> # Add package_derivations_by_guix_revision_range
Add a new channel-news module, along with tables the relevant data
2019-11-16 08:55:54 +00:00
channel_news_tables 2019-11-15T07:32:07Z Christopher Baines <mail@cbaines.net> # Add tables to store channel news
Add new table to store token seeds for build servers
2019-11-23 11:30:36 +00:00
build_server_token_seeds 2019-11-23T09:26:48Z Christopher Baines <mail@cbaines.net> # Add build_server_token_seeds table
Rework the builds and build_status tables as well as related code Allow for build status information to be submitted by POST request. This required some changes to the builds and build_status tables, as for example, the Cuirass build id may not be available, and the derivation may not be know yet, so just record the derivation file name.
2019-11-24 12:59:09 +00:00
rework_builds 2019-11-23T20:41:20Z Christopher Baines <mail@cbaines.net> # Rework the build tables
Begin to add support for importing narinfo files This commit adds the tables, as well as code to support extracting data from narinfo files.
2019-11-30 10:51:58 +00:00
nar_related_tables 2019-11-29T20:28:19Z Christopher Baines <mail@cbaines.net> # Add nar related tables
Add an index on the derivation_file_name field in the builds table As this helps when finding builds relating to specific derivations.
2019-12-04 22:03:52 +01:00
add_index_on_builds_derivation_file_name 2019-12-03T16:23:55Z <chris@phact> # Add index on builds_derivation_file_name
Add a table to record where narinfo files were fetched from Otherwise it's hard to associated narinfo files to build servers.
2019-12-04 22:04:44 +01:00
narinfo_fetch_record 2019-12-03T20:53:28Z <chris@phact> # Add a table to describe fetching a narinfo
Change nars size to bigint As some nars can be bigger than the maximum size of an int.
2019-12-05 14:09:28 +01:00
change_nars_size_to_bigint 2019-12-04T21:24:21Z <chris@phact> # Change nars.size to bigint
Change nar_urls size to bigint As some nars can be bigger than the size of an int.
2019-12-05 14:09:47 +01:00
change_nar_urls_size_to_bigint 2019-12-04T21:49:07Z <chris@phact> # Change nar_urls.size to bigint
Deduplicate builds and add a unique index Duplicate builds could creep in if the code to create them ran concurrently. I didn't exclude them initially, as I was unsure if there should be such a restriction, but at least for now, Cuirass builds map exactly to a single derivation, so use the same restriction here.
2019-12-05 14:10:09 +01:00
sort_out_duplicate_builds 2019-12-05T12:43:53Z <chris@phact> # Sort out duplicate builds
Add some database indexes
2019-12-05 17:19:04 +01:00
add_some_database_indexes 2019-12-05T15:53:04Z Christopher Baines <mail@cbaines.net> # Add some indexes
Start storing and identifying sets of derivation outputs Derivations are effectively equivalent if they produce the same set of outputs, which is possible because of the equivalence of fixed output derivations. A fixed output derivation can be different, but equivalent, because it produces the same fixed output. To better allow tracking equivalent derivations, primarily to allow working out what derivations might correspond to a build, store the sets of derivation outputs, and which derivations they relate to.
2019-12-07 19:21:02 +01:00
derivation_output_sets 2019-12-05T23:19:05Z Christopher Baines <mail@cbaines.net> # Describe the sets of derivation outputs
Add new derivation_output_details_set_id column to the builds table As this will hopefully provide a faster way of associating derivations with builds.
2019-12-07 19:27:31 +01:00
builds_add_derivation_output_details_set_id 2019-12-07T18:25:38Z Christopher Baines <mail@cbaines.net> # Add a derivation_output_details_set_id column to builds
Create an index for the hash component of derivation filenames This is to aid rendering of narinfo files. They're requested with the path /HASH.narinfo, so to quickly find the relevant derivation, this index can be used.
2019-12-26 09:28:20 +00:00
derivations_hash_index 2019-12-24T22:54:19Z Christopher Baines <mail@cbaines.net> # Add index on derivations for the hash component of the filename
Add a new table to nars for derivation source files This will allow serving the nars for derivation source files.
2019-12-28 22:47:58 +00:00
add_derivation_source_file_nars 2019-12-28T20:37:06Z Christopher Baines <mail@cbaines.net> # Add table for derivation source file nars
Create an index on the hash component of the store path For derivation_source_files.
2019-12-29 17:55:04 +00:00
derivation_source_files_store_path_hash_index 2019-12-29T17:53:08Z Christopher Baines <mail@cbaines.net> # Add index on the hash part of the derivation source files store path
Add a table to configure which build servers build what
2020-01-05 12:16:21 +00:00
build_servers_build_config 2020-01-05T12:06:13Z Christopher Baines <mail@cbaines.net> # Add build_servers_build_config table
Don't hardcode the expected x-git-repo header value Rather than expecting it always to be "guix", store the expected value in the database, and use the value of the header to find the relevant repository.
2020-01-11 17:25:08 +00:00
git-repositories-x-git-repo-header 2020-01-11T16:39:32Z Christopher Baines <mail@cbaines.net> # Add x_git_repo_header to git_repositories
Store information about system tests
2020-02-02 20:30:07 +01:00
system_test_tables 2020-02-02T11:36:20Z Christopher Baines <mail@cbaines.net> # Add tables for storing system tests
Rework cross derivations support Stop using the system values as targets, and remove package derivation entries where this is the case. Switch the non-cross derivation case to have a target of "", rather than matching the system, as this makes more sense, and is more consistent now that the target values no longer match the system values. Hardcode some more correct target values, and use these instead. Hopefully this can be better integrated with Guix in the future. This commit also includes a migration attempting to shrink some indexes.
2020-02-08 11:16:08 +00:00
change_package_derivations_target 2020-02-07T19:29:50Z Christopher Baines <mail@cbaines.net> # Change package_derivations target
remove_old_cross_derivations 2020-02-07T19:42:54Z Christopher Baines <mail@cbaines.net> # Remove old cross derivations
increase_fillfactor_for_some_indexes 2020-02-07T20:49:17Z Christopher Baines <mail@cbaines.net> # Increase the fillfactor for some btree indexes
change_package_derivations_by_guix_revision_range_target 2020-02-08T10:13:07Z Christopher Baines <mail@cbaines.net> # Change the values for package_derivations_by_guix_revision_range target
Reference in a new issue Copy permalink