github-mirror/CoreProtect
1
0 Fork 1
mirror of https://github.com/PlayPro/CoreProtect.git synced 2026-10-28 09:44:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Added columnar storage options

Browse Source
This commit is contained in:
Intelli committed 2026-07-16 14:11:40 -06:00
1 parent 130bb803e3
commit 6731831283
152 files changed
+9980 -2186
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+1 -1
View File
@@ -65,7 +65,7 @@ To support the project and obtain a donation key, visit [coreprotect.net/donate]
* Rollbacks and restores with lag-free processing.
* Multi-threaded to ensure server performance is never impacted.
* No configuration required. Install the plugin, and you're good to go.
* SQLite storage by default, with optional MySQL support.
* Embedded DuckDB columnar storage by default, with optional ClickHouse, SQLite, and MySQL support.
* Supports Bukkit, Spigot, Paper, Folia, MultiPaper, and more.
* Permission system support and advanced lookup permissions.
* Easy-to-use commands and inspector.
docs/auto-purge.md
+5 -3
View File
@@ -2,7 +2,7 @@
Automatic purging removes old CoreProtect data on a daily schedule, helping keep database growth under control without requiring manual `/co purge` runs.
> **Note:** Automatic purging is not enabled by default. This feature is exclusive to CoreProtect 24.0+ [Patreon builds](http://patreon.com/coreprotect).
> **Note:** Automatic purging is not enabled by default. This feature is exclusive to CoreProtect 24.0+ [Patreon builds](https://patreon.com/coreprotect).
## Configuration
@@ -29,9 +29,9 @@ When automatic purging is enabled, CoreProtect logs the next scheduled run when
## How It Works
Automatic purging runs in the background and removes old data incrementally in small chunks, with short pauses between database work. The server can continue to be used normally while it runs.
Automatic purging runs in the background. SQLite, MySQL, and DuckDB remove old rows incrementally in small chunks with short pauses between database work. ClickHouse uses its columnar retention path, dropping fully covered monthly partitions for an unfiltered time purge and synchronously removing rows from any partially covered partition.
Automatic purging uses the same CoreProtect data tables as manual purging, but it does not rebuild the SQLite database or optimize MySQL tables. It deletes old rows in place, which helps maintain the current database size over time but may not immediately reduce the database file size or table size on disk.
Automatic purging does not rebuild SQLite, optimize MySQL, or run a ClickHouse `OPTIMIZE FINAL`. DuckDB performs a checkpoint after cleanup. Deleting rows helps control future growth, but it may not immediately reduce the database file or table size on disk.
Only one automatic purge can run at a time. If the server shuts down, a manual purge starts, a database migration or conversion starts, or the consumer is manually paused, the automatic purge stops safely and can continue during the next scheduled run.
@@ -59,6 +59,8 @@ For MySQL, add `#optimize` if you want to reclaim disk space during the initial
This lets the manual purge reduce the existing database size, while automatic purging helps keep the database from growing beyond the configured retention period afterward.
ClickHouse normally reclaims complete old partitions without `#optimize`. A manual ClickHouse purge accepts `#optimize`, but it runs the expensive `OPTIMIZE TABLE ... FINAL` operation and is generally unnecessary. DuckDB checkpoints automatically after a manual or automatic purge; `#optimize` has no additional effect for DuckDB.
## Status
Use `/co status` to see how many rows have been automatically purged since the last server restart.
docs/commands.md
+19 -12
View File
@@ -16,7 +16,7 @@ ___
| [/co reload](#co-reload) | Reload the configuration file |
| [/co status](#co-status) | View the plugin status |
| [/co consumer](#co-consumer) | Toggle consumer processing |
| [/co migrate-db](#co-migrate-db) | Migrate between SQLite and MySQL |
| [/co migrate-db](#co-migrate-db) | Migrate between database backends |
### Alias Commands
@@ -109,11 +109,16 @@ For example, `/co purge t:30d r:#world_nether` will delete all data older than o
You can optionally specify block types in CoreProtect v23+.
For example, `/co purge t:30d i:stone,dirt` will delete all stone and dirt data older than one month, without removing other block data.
**MySQL Optimization**
In CoreProtect v2.15+, adding "#optimize" to the end of the command (e.g. `/co purge t:30d #optimize`) will also optimize your tables and reclaim disk space.
This option is only available when using MySQL, as SQLite purges do this by default.
*Please note adding the #optimize option will significantly slow down your purge, and is generally unnecessary.*
**Database Optimization**
In CoreProtect v2.15+, adding `#optimize` to the end of the command (for example, `/co purge t:30d #optimize`) will also optimize supported database tables and reclaim unused disk space. How this option is handled depends on the database backend:
* SQLite already rebuilds the database from retained data and reclaims unused file space as part of a manual purge, so `#optimize` is not needed.
* MySQL normally deletes matching rows. Adding `#optimize` also optimizes its tables to reclaim unused space.
* DuckDB deletes matching rows in one transaction and checkpoints afterward. `#optimize` has no additional effect.
* ClickHouse drops fully covered monthly partitions for an unfiltered time purge and synchronously removes rows from partial or filtered partitions. Adding `#optimize` also runs `OPTIMIZE TABLE ... FINAL`.
`#optimize` can significantly slow MySQL and ClickHouse purges and is generally unnecessary.
___
@@ -130,17 +135,19 @@ Console command to pause or resume consumer queue processing.
___
### /co migrate-db
Migrate your CoreProtect database between SQLite and MySQL. Console-only command.
Migrate data from the active database backend to a different backend. This is a console-only command.
| Command | Parameters |
| --- | --- |
| /co migrate-db | `<sqlite|mysql>` |
| /co migrate-db | `<sqlite|mysql|duckdb|clickhouse>` |
The target namespace must contain no CoreProtect data; a DuckDB target must use a new database file, and `database-lock` must remain enabled. After a successful migration, CoreProtect automatically updates `database-type` in `config.yml` before queued writes resume.
> **Note:** This feature is exclusive to CoreProtect 23.0+ Patreon builds for Patron supporters.
> **Note:** Migrations between SQLite and MySQL require a CoreProtect 23.0+ Patreon build. Any migration involving DuckDB or ClickHouse requires CoreProtect 25.0+.
For complete migration instructions, safety guidelines, and troubleshooting information, see the [Database Migration documentation](/database-migration/).
___
For complete migration instructions, safety guidelines, and troubleshooting information, see the [Database Migration documentation](/database-migration/).
___
## Parameter Details
docs/config.md
+49
View File
@@ -2,6 +2,55 @@
The CoreProtect configuration file can be found within the CoreProtect folder, at `config.yml`.
## Database Storage
New installations use embedded DuckDB by default. Upgrading an existing SQLite or MySQL installation does not switch it to DuckDB: if `database-type` is absent, CoreProtect derives it from the existing legacy `use-mysql` setting and automatically writes the matching `sqlite` or `mysql` value. No manual change is required. Once present, `database-type` takes precedence over `use-mysql`, and changes take effect after `/co reload` or a restart.
| Value | Storage | Requirements |
| --- | --- | --- |
| `duckdb` | Embedded columnar database; the default for new installations | CoreProtect 25.0+ |
| `clickhouse` | External columnar database | CoreProtect 25.0+ and a reachable ClickHouse 25.6+ server |
| `sqlite` | Embedded legacy relational database | None beyond CoreProtect |
| `mysql` | External legacy relational database | A reachable MySQL server |
The server downloads the DuckDB JDBC driver automatically as a plugin library, while the ClickHouse JDBC driver is packaged inside CoreProtect; neither requires manual driver installation. DuckDB includes its database engine, while ClickHouse requires a separate server.
Changing `database-type` selects a separate dataset; it does not copy existing data. CoreProtect 23.0+ Patreon builds can migrate between SQLite and MySQL; any migration involving DuckDB or ClickHouse requires CoreProtect 25.0+. The target namespace must contain no CoreProtect data; DuckDB requires a new target file, and `database-lock` must remain enabled. See the [database migration guide](/database-migration/) for preparation and switchover instructions.
### DuckDB
The default settings are suitable for most servers:
```yaml
database-type: duckdb
duckdb-memory-limit: 512MB
duckdb-threads: 2
duckdb-max-temp-directory-size: 10GB
```
DuckDB stores its data in `plugins/CoreProtect/database.duckdb`; CoreProtect manages it directly, so no DuckDB service is needed. `duckdb-memory-limit` controls the buffer manager, although total native memory can be higher, and `duckdb-threads` limits query threads. `duckdb-max-temp-directory-size` caps spill data without preallocating space or limiting the database file itself. Queries that exceed both memory and available spill space can fail.
On a pristine installation where the CoreProtect data folder contains no existing files, CoreProtect verifies that DuckDB's native library can run before creating the database file. If the operating system, architecture, or native runtime is incompatible, CoreProtect records `database-type: sqlite` in the generated `config.yml`, logs a warning, and starts with SQLite instead. Existing or preconfigured installations and explicitly configured DuckDB databases never fall back automatically; an initialization error stops database startup so CoreProtect cannot silently open a separate empty history.
### ClickHouse
Configure the ClickHouse HTTP endpoint and credentials before selecting it:
```yaml
database-type: clickhouse
table-prefix: co_
clickhouse-host: 127.0.0.1
clickhouse-port: 8123
clickhouse-database: default
clickhouse-username: default
clickhouse-password: ""
clickhouse-tls: false
```
Create the configured database first with a database engine that provides persistent UUID-backed table identities. `Atomic` is the normal choice for self-hosted ClickHouse and is the default there; for example, `CREATE DATABASE coreprotect ENGINE = Atomic`. CoreProtect checks this capability before creating any tables, then creates and validates its prefixed tables and views. The account needs permission to read `system.databases` and `system.tables`, create and read/write the CoreProtect objects, run mutations and partition drops, and drop or truncate purge-target tables. Server-wide database-creation permission is not required; `#optimize` additionally requires table optimization permission.
Only one active CoreProtect installation may use each ClickHouse database and prefix. Keep `database-lock` enabled and preserve `plugins/CoreProtect/.clickhouse-writer` with the installation; never copy it to another active server. Replicated or distributed tables and multiple active writers are not supported.
## Per-World Configuration
If you'd like to modify the logging settings for a specific world, simply do the following:
docs/database-migration.md
+63 -101
View File
@@ -1,145 +1,107 @@
# Database Migration (SQLite ↔ MySQL)
# Database Migration
CoreProtect 23.0+ includes the ability to migrate your database between SQLite and MySQL without losing any data.
CoreProtect 23.0+ can migrate data between SQLite and MySQL while retaining the source history. CoreProtect 25.0+ adds DuckDB and ClickHouse as migration sources and destinations. Changing `database-type` by itself does not migrate any data.
> **Note:** Database migration functionality is exclusive to CoreProtect 23.0+ [Patreon builds](http://patreon.com/coreprotect) for Patron supporters.
> **Note:** Database migration is available only in [Patreon builds](https://patreon.com/coreprotect) for Patron supporters.
## Overview
The `/co migrate-db` command allows you to seamlessly transfer all your CoreProtect data from SQLite to MySQL or vice versa. This is particularly useful when:
* Starting with SQLite and wanting to upgrade to MySQL for better performance
* Moving from MySQL back to SQLite for simpler server setups
* Transferring data between different server configurations
Database migration can be used to move from SQLite or DuckDB to an external database, return to an embedded database for simpler operation, or switch between any of the supported storage engines. The active backend is always the source, and the command argument selects the destination.
## Command Usage
| Command | Parameters | Description |
| --- | --- | --- |
| `/co migrate-db` | `<sqlite|mysql>` | Migrate to the specified database type |
| `/co migrate-db` | `<sqlite|mysql|duckdb|clickhouse>` | Migrate the active database to the selected backend |
**Examples:**
The command must be run from the server console, and the destination must differ from the active source.
* `/co migrate-db mysql` - Migrate from SQLite to MySQL
* `/co migrate-db sqlite` - Migrate from MySQL to SQLite
For example:
> **Console Only:** This command can only be executed from the server console, not from in-game.
```text
co migrate-db duckdb
co migrate-db clickhouse
```
---
## Before You Start
## Migration Process
1. Stop the server and make a normal backup of the source database and `config.yml`.
2. Leave `database-type` set to the source backend and keep `database-lock: true`.
3. Configure the target while the server is stopped, then start the server with the source backend still selected. MySQL and ClickHouse connection settings must be loaded before the migration starts.
4. Use a target namespace that contains no CoreProtect data. Unrelated tables in the same external database are not affected.
5. Verify that the target has enough storage and that its account has permission to create and modify the required tables.
6. Run the migration during a quiet maintenance period, with no purge, rollback, reload, or other database operation in progress.
### Step 1: Preparation
Prepare the target as follows:
**Before starting the migration:**
| Target | Preparation |
| --- | --- |
| SQLite | Move or archive an existing `plugins/CoreProtect/database.db` if it contains CoreProtect data. CoreProtect creates the file and schema when needed. |
| DuckDB | `plugins/CoreProtect/database.duckdb` must be a new database file so its row ID sequences can be initialized from the source. |
| MySQL | Configure the `mysql-*` options and create the configured database and account. The selected table-prefix namespace must contain no CoreProtect data. |
| ClickHouse | Use ClickHouse 25.6 or newer, create the configured database with persistent UUID-backed table identities (`Atomic` is the normal self-hosted choice), create the account, configure the `clickhouse-*` options, and use a table-prefix namespace containing no CoreProtect data. |
1. Ensure your server is running with your current CoreProtect database
2. Configure your new database in `config.yml` (ensure `use-mysql` is updated correctly)
3. **Important:** Do NOT restart your server or use `/co reload` after modifying the config
`table-prefix` is shared by MySQL and ClickHouse. If the source is MySQL or ClickHouse, leave its prefix unchanged; an external target will use that same prefix. If the source is SQLite or DuckDB, you may select the external target prefix before starting the migration session. Embedded SQLite and DuckDB targets always use `co_`.
### Step 2: Execute Migration
## During Migration
#### 1. **Run the migration command from your server console:**
```
co migrate-db <mysql|sqlite>
```
CoreProtect pauses database persistence, waits for active database work to finish, and copies data in bounded batches. The console shows a throttled copy progress bar with the number of successfully written records, overall percentage, current table, transfer rate, and estimated copy time remaining. It also reports each non-empty table as its copy completes. Verification, activation, and the automatic `config.yml` update follow the copy, so the ETA does not include those final stages.
#### 2. **Monitor the progress:**
* The migration will display detailed progress information
* Large databases may take considerable time to complete
* Progress bars and speed indicators will show current status
New events logged while migration is running remain in the in-memory consumer queue. After successful activation, they are written to the target; if migration fails and the source is restored, persistence resumes there instead. Queued events are not stored for recovery after a process restart, and database-backed commands may report that the database is busy. Large migrations can take hours and use significant CPU, memory, disk, and network bandwidth, so keep server activity low and do not stop or restart the server until the command reports success or failure.
#### 3. **Do not interrupt the process:**
* Avoid restarting your server during migration
* Let the process complete naturally
Do not run `/co reload`, start a purge or rollback, or otherwise change database configuration while migration is running. CoreProtect normally rejects these conflicting operations, but they should not be used as part of the migration procedure.
### Step 3: Post-Migration
## Data Integrity
**After successful completion:**
The migration copies all logical CoreProtect history and reference data, including rollback state and entity tracking. It preserves copied row IDs. MySQL, DuckDB, and ClickHouse targets also preserve allocator high-water marks; SQLite continues after the largest surviving row ID, matching its existing allocation behavior.
1. **Automatic switchover:** CoreProtect will automatically begin using the new database
2. **Verify configuration:** Ensure `use-mysql` is set correctly in your `config.yml` as per step 1
3. **Test functionality:** Perform basic CoreProtect operations to verify everything works
4. **Clean up:** Once satisfied, you may manually delete your old database files
CoreProtect compares table statistics and every copied data row before activation. Database-lock metadata is initialized separately; ClickHouse targets also initialize their own current version row. For a ClickHouse source, the current logical version of each row is copied; superseded internal ClickHouse revisions are storage-engine history rather than separate CoreProtect records.
---
Source history rows are not deleted or replaced. Migration may update operational metadata such as database-lock heartbeats and allocator state.
## Important Considerations
## After a Successful Migration
### Migration Safety
1. CoreProtect atomically changes `database-type` in `config.yml` to the target before queued writes resume. If the legacy `use-mysql` option exists, it is updated at the same time (`true` only for MySQL). Comments and unrelated settings are preserved.
2. The running server immediately begins persisting queued and new events to the target. Keep the server in maintenance mode until the restart check below succeeds.
3. Review the completion message and server log, then test `/co status` and a representative lookup.
4. Restart once and verify that CoreProtect reconnects to the same target and can still read old and new records. No manual database selector edit is required.
5. Resume normal activity and test a newly logged change.
6. Keep the source backup until the migrated database has been verified. Archive or remove it only when you are satisfied that the migration is complete.
* **Non-destructive process:** The migration does not modify or delete your existing database
* **Interrupted migration:** If something goes wrong, simply delete the new database and continue using the old one
* **Data verification:** The process includes automatic verification to ensure data integrity
When either the source or target is ClickHouse, preserve `plugins/CoreProtect/.clickhouse-writer` with the installation so the retained ClickHouse dataset remains recoverable. Do not copy it to another active server or replace it after migration.
### Performance & Requirements
## Failure and Recovery
* **Processing time:** Large databases with millions of records may take hours to migrate
* **Resource usage:** The migration process is resource-intensive and may impact server performance
* **Parallel processing:** SQLite to MySQL migrations use optimized parallel processing for better speed
The command does not resume a partial migration. A failed or interrupted attempt can leave rows, schema objects, or allocator state in the target, so clean only the failed target namespace before retrying. For DuckDB, stop the server and remove the failed target file before starting over. Never clean or replace the source database as part of retrying.
### Restrictions & Limitations
* **Console only:** Cannot be executed from in-game chat
* **Interruption handling:** If interrupted, the target database must be manually wiped before restarting
* **Patreon exclusive:** Only available in CoreProtect 23.0+ [Patreon builds](http://patreon.com/coreprotect)
---
If target activation or the atomic `config.yml` update fails after verification, CoreProtect restores the source when possible and does not resume writes on the target. The selector is changed only after the target has initialized successfully, so an interruption before that change restarts on the source, while an interruption after it restarts on the verified target. If neither the target nor source can be initialized, persistence is halted to avoid writing to an unknown database state. Correct the connection or configuration problem and restart the server; any events still queued only in memory cannot survive that restart.
## Troubleshooting
### Common Issues
### Migration Does Not Start
**Migration won't start:**
* Confirm that the command is being run from the console on a CoreProtect 23.0+ Patreon build, or 25.0+ if DuckDB or ClickHouse is involved.
* Confirm that the selected target differs from the active source backend.
* Keep `database-lock` enabled and wait for any purge, rollback, reload, or previous migration to finish.
* Check that the target namespace contains no CoreProtect data. DuckDB must use a new database file.
* Verify you're using a CoreProtect 23.0+ [Patreon build](http://patreon.com/coreprotect)
* Ensure you're not trying to migrate to the same database type
* Check that no other CoreProtect operations are running
### Connection or Schema Failure
**Migration interrupted:**
* Confirm that the target settings were loaded while `database-type` still selected the source.
* Verify the target host, port, database, credentials, table prefix, and available storage.
* Confirm that the target account can create, read, insert, update or mutate, and delete or drop the required objects.
* For ClickHouse, confirm the server is version 25.6 or newer and that this installation owns the matching `.clickhouse-writer` identity.
* Manually delete/wipe the target database
* Verify server stability before restarting migration
* Consider migrating during low-activity periods
### Verification Failure or Interruption
**Performance issues:**
* Review the server log for the table and row reported by the failure.
* Keep the source database and backup intact.
* Clean the partial target namespace, use a new DuckDB file when applicable, and retry during a quieter maintenance period.
* Monitor server resources during migration
* Consider temporarily reducing server activity
* Large tables may cause temporary slowdowns
### Automatic Configuration Update Fails
**Data verification failures:**
Check that `plugins/CoreProtect/config.yml` is a regular, writable file on a filesystem that supports atomic replacement. CoreProtect reports the migration as failed and restores the source before persistence resumes. The verified target still contains the copied data, so clean only that failed target namespace before retrying; do not manually select it or delete the source.
* Check database connectivity and permissions
* Review server logs for specific error messages
* Ensure sufficient disk space on target database
## Getting Help
### Getting Help
If you encounter issues with database migration:
1. Check server logs for detailed error messages
2. Verify database permissions and connectivity
3. Ensure adequate system resources (RAM, CPU, disk space)
4. Contact support through the CoreProtect Discord with specific error details
---
## Technical Details
The migration process includes several advanced features:
* Batch processing with dynamic sizing based on performance
* Automatic retry mechanisms for temporary failures
* Data integrity verification comparing source and target records
* Progress tracking with estimated completion times
* Graceful error handling with detailed logging
The migration handles all CoreProtect data types including:
* Block changes and rollbacks
* Container transactions
* Player interactions and sessions
* Chat messages and commands
* User data and statistics
* All metadata and configuration data
If a migration fails, keep both databases intact and review the complete error and surrounding server log. When asking for help in the [CoreProtect Discord](https://discord.gg/b4DZ4jy), include your CoreProtect version, source and destination database types, the complete migration error, and the relevant log excerpt.
docs/index.md
+1 -1
View File
@@ -21,7 +21,7 @@ A list of all language codes that can be used with CoreProtect.
Log your own block changes, perform lookups, rollbacks, restores, and more.
[**Database Migration**](/database-migration/)
Migrate your CoreProtect database between SQLite and MySQL.
Migrate your CoreProtect database between supported backends.
[**Automatic Purging**](/auto-purge/)
Automatically remove old CoreProtect data on a daily schedule.
lang/en.yml
+15 -1
View File
@@ -17,11 +17,15 @@ CONSUMER_TOGGLED: "Consumer queue processing has been {paused|resumed}."
CONTAINER_HEADER: "Container Transactions"
CPU_CORES: "CPU cores."
DATABASE_BUSY: "Database busy. Please try again later."
DATABASE_FALLBACK: "Unable to load {0}. Using {1}; config.yml updated."
DATABASE_INDEX_ERROR: "Unable to validate database indexes."
DATABASE_LOCKED_1: "Database locked. Waiting up to 15 seconds..."
DATABASE_LOCKED_2: "Database is already in use. Please try again."
DATABASE_LOCKED_3: "To disable database locking, set \"database-lock: false\"."
DATABASE_LOCKED_4: "Disabling database locking can result in data corruption."
DATABASE_PERSISTENCE_HALTED: "Database writes halted. Please restart the server."
DATABASE_QUEUE_LOSS: "Queued data will be lost when the server stops."
DATABASE_TYPE_OVERRIDE: "Option \"database-type\" overrides \"use-mysql\"."
DATABASE_UNREACHABLE: "Database is unreachable. Discarding data and shutting down."
DEVELOPMENT_BRANCH: "Development branch detected, skipping patch scripts."
DIRT_BLOCK: "Placed a temporary safety block under you."
@@ -36,10 +40,12 @@ FIRST_VERSION: "Initial DB: {0}"
GLOBAL_LOOKUP: "Don't specify a radius to do a global lookup."
GLOBAL_ROLLBACK: "Use \"{0}\" to do a global {rollback|restore}"
HELP_ACTION_1: "Restrict the lookup to a certain action."
HELP_ACTION_2: "Examples: [a:block], [a:+block], [a:-block] [a:click], [a:container], [a:inventory], [a:item], [a:kill], [a:chat], [a:command], [a:sign], [a:session], [a:username]"
HELP_ACTION_2: "Examples: [a:block], [a:+block], [a:-block] [a:click], [a:container], [a:inventory], [a:item], [a:kill], [a:spawn], [a:chat], [a:command], [a:sign], [a:session], [a:username]"
HELP_COMMAND: "Display more info for that command."
HELP_EXCLUDE_1: "Exclude blocks/users."
HELP_EXCLUDE_2: "Examples: [e:stone], [e:Notch], [e:stone,Notch]"
HELP_FILTER_1: "Filter chat, command, or sign text."
HELP_FILTER_2: "Examples: [a:command f:/co], [a:sign f:Shop]"
HELP_HEADER: "{0} Help"
HELP_INCLUDE_1: "Include specific blocks/entities."
HELP_INCLUDE_2: "Examples: [i:stone], [i:zombie], [i:stone,wood,bedrock]"
@@ -113,6 +119,10 @@ LOGGING_ITEMS: "{0} items left to log. Please wait..."
LOGGING_TIME_LIMIT: "Logging time limit reached. Discarding data and shutting down."
LOOKUP_BLOCK: "{0} {placed|broke} {1}."
LOOKUP_CONTAINER: "{0} {added|removed} {1} {2}."
LOOKUP_ENTITY_INTERACTION: "{0} {clicked|sheared|leashed|unleashed} {1}."
LOOKUP_ENTITY_INTERACTION_ORIGIN: "Origin location"
LOOKUP_ENTITY_ORIGIN: "Spawn location"
LOOKUP_ENTITY_SPAWN: "{0} spawned {1}."
LOOKUP_HEADER: "{0} Lookup Results"
LOOKUP_INTERACTION: "{0} {clicked|killed} {1}."
LOOKUP_ITEM: "{0} {picked up|dropped} {1} {2}."
@@ -125,6 +135,7 @@ LOOKUP_STORAGE: "{0} {deposited|withdrew} {1} {2}."
LOOKUP_TIME: "{0} ago"
LOOKUP_USERNAME: "{0} logged in as {1}."
MAXIMUM_RADIUS: "The maximum {lookup|rollback|restore} radius is {0}."
MESSAGE_FILTER_TOO_SHORT: "Minimum message filter length is {0} characters."
MISSING_ACTION_USER: "To use that action, please specify a user."
MISSING_LOOKUP_TIME: "Please specify the amount of time to {lookup|rollback|restore}."
MISSING_LOOKUP_USER: "Please specify a user or {block|radius} to lookup."
@@ -170,6 +181,7 @@ PURGE_ROWS: "{0} {row|rows} of data deleted."
PURGE_STARTED: "Data purge started on \"{0}\"."
PURGE_SUCCESS: "Data purge successful."
RAM_STATS: "{0}GB / {1}GB RAM"
RELOAD_FAILED: "Database reload failed. Please try again."
RELOAD_STARTED: "Reloading configuration - please wait."
RELOAD_SUCCESS: "Configuration successfully reloaded."
ROLLBACK_ABORTED: "Rollback or restore aborted."
@@ -190,6 +202,7 @@ SIGN_HEADER: "Sign Messages"
STATUS_AUTO_PURGE: "Cleanup: {0} {row|rows} auto purged since restart."
STATUS_CONSUMER: "Consumer: {0} {item|items} in queue."
STATUS_DATABASE: "Database: Using {0}."
STATUS_DATABASE_STATE: "({writes halted|unreachable})"
STATUS_INTEGRATION: "{0}: Integration {enabled|disabled}."
STATUS_LICENSE: "License: {0}"
STATUS_SYSTEM: "System: {0}"
@@ -211,6 +224,7 @@ UPDATE_NOTICE: "Notice: {0} is now available."
UPGRADE_IN_PROGRESS: "Upgrade in progress. Please try again later."
USER_NOT_FOUND: "User \"{0}\" not found."
USER_OFFLINE: "The user \"{0}\" is not online."
USING_DATABASE: "Using {0} for data storage."
USING_MYSQL: "Using MySQL for data storage."
USING_SQLITE: "Using SQLite for data storage."
VALID_DONATION_KEY: "Valid donation key."
pom.xml
+29 -4
View File
@@ -7,8 +7,9 @@
<project.branch></project.branch>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<skipTests>true</skipTests>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<maven.compiler.release>11</maven.compiler.release>
<clickhouse.version>0.9.8</clickhouse.version>
<duckdb.version>1.4.5.0</duckdb.version>
</properties>
<build>
<resources>
@@ -28,8 +29,7 @@
<artifactId>maven-compiler-plugin</artifactId>
<version>3.15.0</version>
<configuration>
<source>11</source>
<target>11</target>
<release>${maven.compiler.release}</release>
</configuration>
<executions>
<execution>
@@ -51,6 +51,7 @@
<configuration>
<archive>
<manifestEntries>
<Multi-Release>true</Multi-Release>
<paperweight-mappings-namespace>mojang</paperweight-mappings-namespace>
</manifestEntries>
</archive>
@@ -250,6 +251,24 @@
<artifactId>HikariCP</artifactId>
<version>7.0.2</version>
</dependency>
<dependency>
<groupId>org.duckdb</groupId>
<artifactId>duckdb_jdbc</artifactId>
<version>${duckdb.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.clickhouse</groupId>
<artifactId>clickhouse-jdbc</artifactId>
<version>${clickhouse.version}</version>
<classifier>all</classifier>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
@@ -309,6 +328,12 @@
<version>3.53.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.mysql</groupId>
<artifactId>mysql-connector-j</artifactId>
<version>9.6.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
src/main/java/net/coreprotect/CoreProtectAPI.java
+1 -1
View File
@@ -1101,7 +1101,7 @@ public class CoreProtectAPI extends Queue {
}
}
else { // only print exception on development branch
ErrorReporter.report(new IllegalStateException("API rollback skipped: database purge in progress"), ConfigHandler.EDITION_BRANCH.contains("-dev"));
ErrorReporter.report(new IllegalStateException("API rollback skipped: another database operation is active or persistence is halted"), ConfigHandler.EDITION_BRANCH.contains("-dev"));
}
}
else {
src/main/java/net/coreprotect/api/BlockAPI.java
+13 -22
View File
@@ -17,6 +17,7 @@ import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.Database;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.utility.BlockUtils;
import net.coreprotect.utility.DatabaseUtils;
import net.coreprotect.utility.StringUtils;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.ErrorReporter;
@@ -71,7 +72,7 @@ public class BlockAPI {
}
try (Statement statement = connection.createStatement()) {
String query = "SELECT time,user,action,type,data,blockdata,rolled_back FROM " + ConfigHandler.prefix + "block " + WorldUtils.getWidIndex("block") + "WHERE wid = '" + worldId + "' AND x = '" + x + "' AND z = '" + z + "' AND y = '" + y + "' AND time > '" + checkTime + "' ORDER BY rowid DESC";
String query = "SELECT time," + ConfigHandler.databaseType.getUserColumn() + ",action,type,data,blockdata,rolled_back FROM " + ConfigHandler.prefix + "block " + WorldUtils.getWidIndex("block") + "WHERE wid = " + worldId + " AND x = " + x + " AND z = " + z + " AND y = " + y + " AND time > " + checkTime + " ORDER BY rowid DESC";
try (ResultSet results = statement.executeQuery(query)) {
while (results.next()) {
@@ -80,14 +81,10 @@ public class BlockAPI {
String resultAction = results.getString("action");
int resultType = results.getInt("type");
String resultData = results.getString("data");
byte[] resultBlockData = results.getBytes("blockdata");
byte[] resultBlockData = DatabaseUtils.getBytes(results, "blockdata");
String resultRolledBack = results.getString("rolled_back");
if (ConfigHandler.playerIdCacheReversed.get(resultUserId) == null) {
UserStatement.loadName(connection, resultUserId);
}
String resultUser = ConfigHandler.playerIdCacheReversed.get(resultUserId);
String resultUser = UserStatement.getName(connection, resultUserId);
String blockData = BlockUtils.byteDataToString(resultBlockData, resultType);
String[] lookupData = new String[] { resultTime, resultUser, String.valueOf(x), String.valueOf(y), String.valueOf(z), String.valueOf(resultType), resultData, resultAction, resultRolledBack, String.valueOf(worldId), blockData };
@@ -149,15 +146,15 @@ public class BlockAPI {
String worldName = block.getWorld().getName();
int worldId = WorldUtils.getWorldId(worldName);
StringBuilder query = new StringBuilder("SELECT time,user,action,type,data,blockdata,rolled_back,wid,x,y,z FROM ");
StringBuilder query = new StringBuilder("SELECT time," + ConfigHandler.databaseType.getUserColumn() + ",action,type,data,blockdata,rolled_back,wid,x,y,z FROM ");
query.append(ConfigHandler.prefix).append("block ").append(WorldUtils.getWidIndex("block"));
query.append("WHERE wid = ? AND x = ? AND z = ? AND y = ? AND time > ?");
if (userId != null) {
query.append(" AND user = ?");
query.append(" AND ").append(ConfigHandler.databaseType.getUserColumn()).append(" = ?");
}
query.append(" ORDER BY rowid DESC");
if (options.hasLimit()) {
query.append(" LIMIT ").append(options.getLimitOffset()).append(", ").append(options.getLimitCount());
query.append(" LIMIT ").append(options.getLimitCount()).append(" OFFSET ").append(options.getLimitOffset());
}
try (PreparedStatement statement = connection.prepareStatement(query.toString())) {
@@ -232,10 +229,10 @@ public class BlockAPI {
filter.appendEntityContainerWhere(entityWhere, "entity_rows");
StringBuilder query = new StringBuilder("SELECT * FROM (");
query.append("SELECT 0 AS source,container_rows.rowid AS id,container_rows.time,container_rows.user,container_rows.wid,container_rows.x,container_rows.y,container_rows.z,container_rows.action,container_rows.type,container_rows.data,container_rows.amount,container_rows.metadata,container_rows.rolled_back FROM ")
query.append("SELECT 0 AS source,container_rows.rowid AS id,container_rows.time,container_rows.").append(ConfigHandler.databaseType.getUserColumn()).append(",container_rows.wid,container_rows.x,container_rows.y,container_rows.z,container_rows.action,container_rows.type,container_rows.data,container_rows.amount,container_rows.metadata,container_rows.rolled_back FROM ")
.append(ConfigHandler.prefix).append("container container_rows ").append(containerWhere);
query.append(" UNION ALL ");
query.append("SELECT 1 AS source,entity_rows.rowid AS id,entity_rows.time,entity_rows.user,spawn_rows.current_wid AS wid,spawn_rows.x,spawn_rows.y,spawn_rows.z,entity_rows.action,entity_rows.type,entity_rows.data,entity_rows.amount,entity_rows.metadata,entity_rows.rolled_back FROM ")
query.append("SELECT 1 AS source,entity_rows.rowid AS id,entity_rows.time,entity_rows.").append(ConfigHandler.databaseType.getUserColumn()).append(",spawn_rows.current_wid AS wid,spawn_rows.x,spawn_rows.y,spawn_rows.z,entity_rows.action,entity_rows.type,entity_rows.data,entity_rows.amount,entity_rows.metadata,entity_rows.rolled_back FROM ")
.append(ConfigHandler.prefix).append("entity_container entity_rows JOIN ").append(ConfigHandler.prefix).append("entity_spawn spawn_rows ON spawn_rows.rowid=entity_rows.entity_spawn_rowid ").append(entityWhere);
query.append(") AS container_lookup ORDER BY time DESC,source DESC,id DESC");
filter.appendLimit(query);
@@ -260,27 +257,21 @@ public class BlockAPI {
private static ContainerResult parseContainerResult(Connection connection, ResultSet results) throws Exception {
int resultUserId = results.getInt("user");
String resultUser = ConfigHandler.playerIdCacheReversed.get(resultUserId);
if (resultUser == null) {
resultUser = UserStatement.loadName(connection, resultUserId);
}
String resultUser = UserStatement.getName(connection, resultUserId);
return new ContainerResult(
results.getLong("time"), resultUser, WorldUtils.getWorldName(results.getInt("wid")), (int) Math.floor(results.getDouble("x")), (int) Math.floor(results.getDouble("y")), (int) Math.floor(results.getDouble("z")),
results.getInt("type"), results.getInt("data"), results.getInt("amount"), results.getBytes("metadata"),
results.getInt("type"), results.getInt("data"), results.getInt("amount"), DatabaseUtils.getBytes(results, "metadata"),
results.getInt("action"), results.getInt("rolled_back")
);
}
private static BlockResult parseBlockResult(Connection connection, ResultSet results, String worldName) throws Exception {
int resultUserId = results.getInt("user");
String resultUser = ConfigHandler.playerIdCacheReversed.get(resultUserId);
if (resultUser == null) {
resultUser = UserStatement.loadName(connection, resultUserId);
}
String resultUser = UserStatement.getName(connection, resultUserId);
int resultType = results.getInt("type");
String blockData = BlockUtils.byteDataToString(results.getBytes("blockdata"), resultType);
String blockData = BlockUtils.byteDataToString(DatabaseUtils.getBytes(results, "blockdata"), resultType);
return new BlockResult(
results.getLong("time"), resultUser, worldName, results.getInt("x"), results.getInt("y"), results.getInt("z"),
src/main/java/net/coreprotect/api/InventoryAPI.java
+10 -12
View File
@@ -13,6 +13,7 @@ import net.coreprotect.database.Database;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.model.item.InventorySources;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.DatabaseUtils;
import net.coreprotect.utility.ErrorReporter;
/**
@@ -59,8 +60,8 @@ public class InventoryAPI {
parameterIndex = filter.bindEntityContainer(statement, parameterIndex);
parameterIndex = filter.bind(statement, parameterIndex);
if (options.hasLimit()) {
statement.setInt(parameterIndex++, options.getLimitOffset());
statement.setInt(parameterIndex, options.getLimitCount());
statement.setInt(parameterIndex++, options.getLimitCount());
statement.setInt(parameterIndex, options.getLimitOffset());
}
try (ResultSet results = statement.executeQuery()) {
@@ -79,20 +80,20 @@ public class InventoryAPI {
private static String buildQuery(String where, String entityWhere, boolean hasLimit) {
StringBuilder query = new StringBuilder("SELECT * FROM (");
query.append("SELECT 0 AS source,rowid AS id,time,user,wid,x,y,z,type,data,1 AS amount,meta AS metadata,action,rolled_back FROM ")
query.append("SELECT 0 AS source,rowid AS id,time,").append(ConfigHandler.databaseType.getUserColumn()).append(",wid,x,y,z,type,data,1 AS amount,meta AS metadata,action,rolled_back FROM ")
.append(ConfigHandler.prefix).append("block ").append(where).append(" AND action = 1");
query.append(" UNION ALL ");
query.append("SELECT 1 AS source,rowid AS id,time,user,wid,x,y,z,type,data,amount,metadata,action,rolled_back FROM ")
query.append("SELECT 1 AS source,rowid AS id,time,").append(ConfigHandler.databaseType.getUserColumn()).append(",wid,x,y,z,type,data,amount,metadata,action,rolled_back FROM ")
.append(ConfigHandler.prefix).append("container ").append(where);
query.append(" UNION ALL ");
query.append("SELECT ").append(InventorySources.ENTITY_CONTAINER).append(" AS source,entity_rows.rowid AS id,entity_rows.time,entity_rows.user,spawn_rows.current_wid AS wid,spawn_rows.x,spawn_rows.y,spawn_rows.z,entity_rows.type,entity_rows.data,entity_rows.amount,entity_rows.metadata,entity_rows.action,entity_rows.rolled_back FROM ")
query.append("SELECT ").append(InventorySources.ENTITY_CONTAINER).append(" AS source,entity_rows.rowid AS id,entity_rows.time,entity_rows.").append(ConfigHandler.databaseType.getUserColumn()).append(",spawn_rows.current_wid AS wid,spawn_rows.x,spawn_rows.y,spawn_rows.z,entity_rows.type,entity_rows.data,entity_rows.amount,entity_rows.metadata,entity_rows.action,entity_rows.rolled_back FROM ")
.append(ConfigHandler.prefix).append("entity_container entity_rows JOIN ").append(ConfigHandler.prefix).append("entity_spawn spawn_rows ON spawn_rows.rowid=entity_rows.entity_spawn_rowid ").append(entityWhere);
query.append(" UNION ALL ");
query.append("SELECT 2 AS source,rowid AS id,time,user,wid,x,y,z,type,0 AS data,amount,data AS metadata,action,rolled_back FROM ")
query.append("SELECT 2 AS source,rowid AS id,time,").append(ConfigHandler.databaseType.getUserColumn()).append(",wid,x,y,z,type,0 AS data,amount,data AS metadata,action,rolled_back FROM ")
.append(ConfigHandler.prefix).append("item ").append(where);
query.append(") AS inventory_lookup ORDER BY time DESC, source DESC, id DESC");
if (hasLimit) {
query.append(" LIMIT ?, ?");
query.append(" LIMIT ? OFFSET ?");
}
return query.toString();
@@ -100,10 +101,7 @@ public class InventoryAPI {
private static InventoryResult parseInventoryResult(Connection connection, ResultSet results) throws Exception {
int userId = results.getInt("user");
String username = ConfigHandler.playerIdCacheReversed.get(userId);
if (username == null) {
username = UserStatement.loadName(connection, userId);
}
String username = UserStatement.getName(connection, userId);
int source = results.getInt("source");
if (source == InventorySources.ENTITY_CONTAINER) {
@@ -113,7 +111,7 @@ public class InventoryAPI {
return new InventoryResult(
results.getLong("time"), username, WorldUtils.getWorldName(results.getInt("wid")),
(int) Math.floor(results.getDouble("x")), (int) Math.floor(results.getDouble("y")), (int) Math.floor(results.getDouble("z")),
results.getInt("type"), results.getInt("data"), results.getInt("amount"), results.getBytes("metadata"),
results.getInt("type"), results.getInt("data"), results.getInt("amount"), DatabaseUtils.getBytes(results, "metadata"),
results.getInt("action"), results.getInt("rolled_back"), source
);
}
src/main/java/net/coreprotect/api/ItemAPI.java
+4 -6
View File
@@ -13,6 +13,7 @@ import net.coreprotect.database.Database;
import net.coreprotect.database.logger.ItemLogger;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.DatabaseUtils;
import net.coreprotect.utility.ErrorReporter;
/**
@@ -41,7 +42,7 @@ public class ItemAPI {
return result;
}
StringBuilder query = new StringBuilder("SELECT time,user,wid,x,y,z,type,data,amount,action,rolled_back FROM ");
StringBuilder query = new StringBuilder("SELECT time," + ConfigHandler.databaseType.getUserColumn() + ",wid,x,y,z,type,data,amount,action,rolled_back FROM ");
query.append(ConfigHandler.prefix).append("item ");
if (filter.hasLocation()) {
query.append(WorldUtils.getWidIndex("item"));
@@ -75,15 +76,12 @@ public class ItemAPI {
private static ItemResult parseItemResult(Connection connection, ResultSet results) throws Exception {
int userId = results.getInt("user");
String username = ConfigHandler.playerIdCacheReversed.get(userId);
if (username == null) {
username = UserStatement.loadName(connection, userId);
}
String username = UserStatement.getName(connection, userId);
return new ItemResult(
results.getLong("time"), username, WorldUtils.getWorldName(results.getInt("wid")),
results.getInt("x"), results.getInt("y"), results.getInt("z"),
results.getInt("type"), results.getInt("amount"), results.getBytes("data"),
results.getInt("type"), results.getInt("amount"), DatabaseUtils.getBytes(results, "data"),
results.getInt("action"), results.getInt("rolled_back")
);
}
src/main/java/net/coreprotect/api/LookupFilter.java
+3 -3
View File
@@ -60,7 +60,7 @@ final class LookupFilter {
query.append("WHERE ").append(qualifier).append("time > ?");
if (userId != null) {
query.append(" AND ").append(qualifier).append("user = ?");
query.append(" AND ").append(qualifier).append(ConfigHandler.databaseType.getUserColumn()).append(" = ?");
}
if (location != null) {
@@ -79,7 +79,7 @@ final class LookupFilter {
String entity = "current_spawn_rows.";
query.append("WHERE ").append(transaction).append("time > ?");
if (userId != null) {
query.append(" AND ").append(transaction).append("user = ?");
query.append(" AND ").append(transaction).append(ConfigHandler.databaseType.getUserColumn()).append(" = ?");
}
if (location == null) {
return;
@@ -106,7 +106,7 @@ final class LookupFilter {
void appendLimit(StringBuilder query) {
if (limitOffset >= 0 && limitCount >= 0) {
query.append(" LIMIT ").append(limitOffset).append(", ").append(limitCount);
query.append(" LIMIT ").append(limitCount).append(" OFFSET ").append(limitOffset);
}
}
src/main/java/net/coreprotect/api/MessageAPI.java
+4 -25
View File
@@ -11,8 +11,8 @@ import net.coreprotect.config.Config;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.Database;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.ErrorReporter;
import net.coreprotect.utility.WorldUtils;
/**
* Provides API methods for player chat and command lookups.
@@ -55,7 +55,7 @@ public class MessageAPI {
return result;
}
StringBuilder query = new StringBuilder("SELECT time,user,wid,x,y,z,message FROM ");
StringBuilder query = new StringBuilder("SELECT time," + ConfigHandler.databaseType.getUserColumn() + ",wid,x,y,z,message FROM ");
query.append(ConfigHandler.prefix).append(table).append(" ");
if (filter.hasLocation()) {
query.append(WorldUtils.getWidIndex(table));
@@ -86,33 +86,12 @@ public class MessageAPI {
return null;
}
Integer cachedId = ConfigHandler.playerIdCache.get(user.toLowerCase(java.util.Locale.ROOT));
if (cachedId != null) {
return cachedId;
}
String collate = Config.getGlobal().MYSQL ? "" : " COLLATE NOCASE";
try (PreparedStatement statement = connection.prepareStatement("SELECT rowid FROM " + ConfigHandler.prefix + "user WHERE user = ?" + collate + " ORDER BY rowid ASC LIMIT 0, 1")) {
statement.setString(1, user);
try (ResultSet results = statement.executeQuery()) {
if (results.next()) {
int id = results.getInt("rowid");
ConfigHandler.playerIdCache.put(user.toLowerCase(java.util.Locale.ROOT), id);
ConfigHandler.playerIdCacheReversed.put(id, user);
return id;
}
}
}
return -1;
return UserStatement.findId(connection, user);
}
private static MessageResult parseMessageResult(Connection connection, ResultSet results, int actionId) throws Exception {
int userId = results.getInt("user");
String username = ConfigHandler.playerIdCacheReversed.get(userId);
if (username == null) {
username = UserStatement.loadName(connection, userId);
}
String username = UserStatement.getName(connection, userId);
return new MessageResult(
results.getLong("time"), username, WorldUtils.getWorldName(results.getInt("wid")),
src/main/java/net/coreprotect/api/SessionLookup.java
+8 -34
View File
@@ -6,7 +6,6 @@ import java.sql.ResultSet;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import net.coreprotect.api.result.SessionResult;
import net.coreprotect.config.Config;
@@ -66,8 +65,10 @@ public class SessionLookup {
int time = (int) (System.currentTimeMillis() / 1000L);
int checkTime = calculateCheckTime(time, offset);
// Get user ID from cache or load it
int userId = getUserId(connection, user);
Integer userId = MessageAPI.getUserId(connection, user);
if (userId == null || userId < 0) {
return result;
}
// Query session data from database
try (Statement statement = connection.createStatement()) {
@@ -112,7 +113,7 @@ public class SessionLookup {
return result;
}
StringBuilder query = new StringBuilder("SELECT time,user,wid,x,y,z,action FROM ");
StringBuilder query = new StringBuilder("SELECT time," + ConfigHandler.databaseType.getUserColumn() + ",wid,x,y,z,action FROM ");
query.append(ConfigHandler.prefix).append("session ");
if (filter.hasLocation()) {
query.append(WorldUtils.getWidIndex("session"));
@@ -151,26 +152,6 @@ public class SessionLookup {
return (offset > 0) ? currentTime - offset : 0;
}
/**
* Gets the user ID for the specified username.
* If the user ID is not in the cache, it will be loaded from the database.
*
* @param connection
* The database connection
* @param username
* The username to get the ID for
* @return The user ID
*/
private static int getUserId(Connection connection, String username) {
String lowerUsername = username.toLowerCase(Locale.ROOT);
if (ConfigHandler.playerIdCache.get(lowerUsername) == null) {
UserStatement.loadId(connection, username, null);
}
return ConfigHandler.playerIdCache.get(lowerUsername);
}
/**
* Builds the SQL query for retrieving session data.
*
@@ -181,7 +162,7 @@ public class SessionLookup {
* @return The SQL query string
*/
private static String buildSessionQuery(int userId, int checkTime) {
return "SELECT time,user,wid,x,y,z,action FROM " + ConfigHandler.prefix + "session WHERE user = '" + userId + "' AND time > '" + checkTime + "' ORDER BY rowid DESC";
return "SELECT time," + ConfigHandler.databaseType.getUserColumn() + ",wid,x,y,z,action FROM " + ConfigHandler.prefix + "session WHERE " + ConfigHandler.databaseType.getUserColumn() + " = " + userId + " AND time > " + checkTime + " ORDER BY rowid DESC";
}
/**
@@ -206,11 +187,7 @@ public class SessionLookup {
String resultZ = results.getString("z");
String resultAction = results.getString("action");
// Get username from cache or load it
if (ConfigHandler.playerIdCacheReversed.get(resultUserId) == null) {
UserStatement.loadName(connection, resultUserId);
}
String resultUser = ConfigHandler.playerIdCacheReversed.get(resultUserId);
String resultUser = UserStatement.getName(connection, resultUserId);
// Create and return the session data array
return new String[] { resultTime, resultUser, resultX, resultY, resultZ, resultWorldId, type, resultAction };
@@ -218,10 +195,7 @@ public class SessionLookup {
private static SessionResult parseSessionResult(Connection connection, ResultSet results) throws Exception {
int userId = results.getInt("user");
String username = ConfigHandler.playerIdCacheReversed.get(userId);
if (username == null) {
username = UserStatement.loadName(connection, userId);
}
String username = UserStatement.getName(connection, userId);
return new SessionResult(
results.getLong("time"), username, WorldUtils.getWorldName(results.getInt("wid")),
src/main/java/net/coreprotect/api/SignAPI.java
+2 -5
View File
@@ -51,7 +51,7 @@ public class SignAPI {
return result;
}
StringBuilder query = new StringBuilder("SELECT time,user,wid,x,y,z,action,color,color_secondary,data,waxed,face,line_1,line_2,line_3,line_4,line_5,line_6,line_7,line_8 FROM ");
StringBuilder query = new StringBuilder("SELECT time," + ConfigHandler.databaseType.getUserColumn() + ",wid,x,y,z,action,color,color_secondary,data,waxed,face,line_1,line_2,line_3,line_4,line_5,line_6,line_7,line_8 FROM ");
query.append(ConfigHandler.prefix).append("sign ");
if (filter.hasLocation()) {
query.append(WorldUtils.getWidIndex("sign"));
@@ -80,10 +80,7 @@ public class SignAPI {
private static SignResult parseSignResult(Connection connection, ResultSet results) throws Exception {
int userId = results.getInt("user");
String username = ConfigHandler.playerIdCacheReversed.get(userId);
if (username == null) {
username = UserStatement.loadName(connection, userId);
}
String username = UserStatement.getName(connection, userId);
String[] lines = new String[] {
valueOrEmpty(results.getString("line_1")),
src/main/java/net/coreprotect/api/UsernameAPI.java
+10 -14
View File
@@ -6,13 +6,14 @@ import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Set;
import net.coreprotect.api.result.UsernameResult;
import net.coreprotect.config.Config;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.Database;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.utility.DatabaseUtils;
import net.coreprotect.utility.ErrorReporter;
/**
@@ -54,7 +55,7 @@ public class UsernameAPI {
checkTime = (int) (System.currentTimeMillis() / 1000L) - options.getTime();
}
StringBuilder query = new StringBuilder("SELECT time,uuid,user FROM ");
StringBuilder query = new StringBuilder("SELECT time,uuid," + ConfigHandler.databaseType.getUserColumn() + " FROM ");
query.append(ConfigHandler.prefix).append("username_log WHERE time > ?");
if (!uuids.isEmpty()) {
query.append(" AND uuid IN (");
@@ -63,7 +64,7 @@ public class UsernameAPI {
}
query.append(" ORDER BY rowid DESC");
if (options.hasLimit()) {
query.append(" LIMIT ").append(options.getLimitOffset()).append(", ").append(options.getLimitCount());
query.append(" LIMIT ").append(options.getLimitCount()).append(" OFFSET ").append(options.getLimitOffset());
}
try (PreparedStatement statement = connection.prepareStatement(query.toString())) {
@@ -75,7 +76,7 @@ public class UsernameAPI {
try (ResultSet results = statement.executeQuery()) {
while (results.next()) {
result.add(parseUsernameResult(results));
result.add(parseUsernameResult(connection, results));
}
}
}
@@ -103,13 +104,8 @@ public class UsernameAPI {
return result;
}
String cachedUuid = ConfigHandler.uuidCache.get(user.toLowerCase(Locale.ROOT));
if (cachedUuid != null) {
result.add(cachedUuid);
}
String collate = Config.getGlobal().MYSQL ? "" : " COLLATE NOCASE";
try (PreparedStatement statement = connection.prepareStatement("SELECT uuid FROM " + ConfigHandler.prefix + "user WHERE user = ?" + collate)) {
String userMatch = DatabaseUtils.caseInsensitiveEquals(ConfigHandler.databaseType.getUserColumn());
try (PreparedStatement statement = connection.prepareStatement("SELECT uuid FROM " + ConfigHandler.prefix + "user WHERE " + userMatch)) {
statement.setString(1, user);
try (ResultSet results = statement.executeQuery()) {
while (results.next()) {
@@ -118,7 +114,7 @@ public class UsernameAPI {
}
}
try (PreparedStatement statement = connection.prepareStatement("SELECT uuid FROM " + ConfigHandler.prefix + "username_log WHERE user = ?" + collate)) {
try (PreparedStatement statement = connection.prepareStatement("SELECT uuid FROM " + ConfigHandler.prefix + "username_log WHERE " + userMatch)) {
statement.setString(1, user);
try (ResultSet results = statement.executeQuery()) {
while (results.next()) {
@@ -138,10 +134,10 @@ public class UsernameAPI {
return value.length() == 36 && value.charAt(8) == '-' && value.charAt(13) == '-' && value.charAt(18) == '-' && value.charAt(23) == '-';
}
private static UsernameResult parseUsernameResult(ResultSet results) throws Exception {
private static UsernameResult parseUsernameResult(Connection connection, ResultSet results) throws Exception {
String uuid = results.getString("uuid");
String username = results.getString("user");
String player = ConfigHandler.uuidCacheReversed.get(uuid);
String player = UserStatement.getNameByUuid(uuid);
if (player == null) {
player = username;
}
src/main/java/net/coreprotect/command/PurgeCommand.java
+288 -71
View File
@@ -1,11 +1,17 @@
package net.coreprotect.command;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import java.text.NumberFormat;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Locale;
@@ -18,10 +24,10 @@ import org.bukkit.entity.EntityType;
import org.bukkit.entity.Player;
import net.coreprotect.bukkit.BukkitAdapter;
import net.coreprotect.config.Config;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.database.Database;
import net.coreprotect.database.PurgePolicy;
import net.coreprotect.language.Phrase;
import net.coreprotect.language.Selector;
import net.coreprotect.model.action.LookupActions;
@@ -37,7 +43,66 @@ import net.coreprotect.utility.ErrorReporter;
public class PurgeCommand extends Consumer {
public static final List<String> PURGE_TABLES = Arrays.asList("sign", "container", "entity_container", "entity_interaction", "item", "skull", "session", "chat", "command", "entity", "block");
private static final long CONNECTION_DRAIN_TIMEOUT_MILLIS = 60000L;
private static volatile Thread activePurgeThread;
private static volatile Statement activePurgeStatement;
private static volatile boolean shutdownCancellationRequested;
public static void resetShutdownCancellation() {
shutdownCancellationRequested = false;
}
public static void cancelForShutdown() {
shutdownCancellationRequested = true;
Database.cancelClickHousePurge();
Statement statement = activePurgeStatement;
if (statement != null) {
try {
statement.cancel();
}
catch (Exception ignored) {
}
}
Thread thread = activePurgeThread;
if (thread != null) {
thread.interrupt();
}
}
public static boolean isPurgeWorkerRunning() {
Thread thread = activePurgeThread;
return thread != null && thread.isAlive();
}
private static void requirePurgeNotCancelled() throws InterruptedException {
if (shutdownCancellationRequested || Thread.currentThread().isInterrupted()) {
throw new InterruptedException("Purge interrupted during shutdown");
}
}
private static PreparedStatement preparePurgeStatement(Connection connection, String query) throws SQLException, InterruptedException {
requirePurgeNotCancelled();
PreparedStatement statement = connection.prepareStatement(query);
activePurgeStatement = statement;
requirePurgeNotCancelled();
return statement;
}
private static Statement createPurgeStatement(Connection connection) throws SQLException, InterruptedException {
requirePurgeNotCancelled();
Statement statement = connection.createStatement();
activePurgeStatement = statement;
requirePurgeNotCancelled();
return statement;
}
private static void reportPurgeFailure(Exception exception) throws Exception {
if (shutdownCancellationRequested || Thread.currentThread().isInterrupted()) {
throw exception;
}
ErrorReporter.report(exception);
}
private static String findUnsupportedPurgeArgument(String[] args) {
boolean includeContinuation = false;
@@ -153,6 +218,7 @@ public class PurgeCommand extends Consumer {
StringBuilder restrict = new StringBuilder();
String includeBlock = "";
List<Integer> includeBlockIds = new ArrayList<>();
String includeEntity = "";
boolean hasBlock = false;
boolean item = false;
@@ -168,17 +234,20 @@ public class PurgeCommand extends Consumer {
if (restrictTarget instanceof Material) {
targetName = ((Material) restrictTarget).name();
int blockId = MaterialUtils.getBlockId(targetName, false);
includeBlockIds.add(blockId);
if (includeListMaterial.length() == 0) {
includeListMaterial = includeListMaterial.append(MaterialUtils.getBlockId(targetName, false));
includeListMaterial = includeListMaterial.append(blockId);
}
else {
includeListMaterial.append(",").append(MaterialUtils.getBlockId(targetName, false));
includeListMaterial.append(",").append(blockId);
}
/* Include legacy IDs */
int legacyId = BukkitAdapter.ADAPTER.getLegacyBlockId((Material) restrictTarget);
if (legacyId > 0) {
includeListMaterial.append(",").append(legacyId);
includeBlockIds.add(legacyId);
}
targetName = ((Material) restrictTarget).name().toLowerCase(Locale.ROOT);
@@ -199,6 +268,7 @@ public class PurgeCommand extends Consumer {
}
else if (restrictTarget instanceof String) {
int blockId = MaterialUtils.getBlockId((String) restrictTarget, false);
includeBlockIds.add(blockId);
if (includeListMaterial.length() == 0) {
includeListMaterial = includeListMaterial.append(blockId);
}
@@ -239,6 +309,7 @@ public class PurgeCommand extends Consumer {
final StringBuilder restrictTargets = restrict;
final String includeBlockFinal = includeBlock;
final List<Integer> includeBlockIdsFinal = List.copyOf(includeBlockIds);
final boolean optimize = optimizeCheck;
final boolean hasBlockRestriction = hasBlock;
final int restrictCountFinal = restrictCount;
@@ -249,14 +320,27 @@ public class PurgeCommand extends Consumer {
public void run() {
boolean purgeClaimed = false;
boolean consumerPaused = false;
boolean duckTransaction = false;
boolean duckPurgeStarted = false;
boolean duckRollbackSucceeded = false;
boolean maintenanceLocked = false;
boolean resumePersistence = true;
boolean handoffStarted = false;
Connection connection = null;
Statement transactionStatement = null;
try {
requirePurgeNotCancelled();
if (Consumer.isPersistenceHalted()) {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.DATABASE_BUSY));
return;
}
long timestamp = (System.currentTimeMillis() / 1000L);
long timeStart = startTime > 0 ? (timestamp - startTime) : 0;
long timeEnd = timestamp - endTime;
long removed = 0;
Connection connection = null;
for (int i = 0; i <= 5; i++) {
requirePurgeNotCancelled();
connection = Database.getConnection(false, 500);
if (connection != null) {
break;
@@ -269,9 +353,12 @@ public class PurgeCommand extends Consumer {
return;
}
requirePurgeNotCancelled();
Consumer.OperationStartResult startResult = Consumer.claimPurge();
if (startResult != Consumer.OperationStartResult.STARTED) {
Phrase phrase = startResult == Consumer.OperationStartResult.PURGE_RUNNING ? Phrase.PURGE_IN_PROGRESS : Phrase.ROLLBACK_IN_PROGRESS;
Phrase phrase = startResult == Consumer.OperationStartResult.PURGE_RUNNING ? Phrase.PURGE_IN_PROGRESS
: startResult == Consumer.OperationStartResult.RELOAD_RUNNING ? Phrase.DATABASE_BUSY
: startResult == Consumer.OperationStartResult.PERSISTENCE_HALTED ? Phrase.DATABASE_PERSISTENCE_HALTED : Phrase.ROLLBACK_IN_PROGRESS;
Chat.sendGlobalMessage(player, Phrase.build(phrase));
try {
connection.close();
@@ -282,6 +369,8 @@ public class PurgeCommand extends Consumer {
return;
}
purgeClaimed = true;
activePurgeThread = Thread.currentThread();
requirePurgeNotCancelled();
if (argWid > 0) {
String worldName = CommandParser.parseWorldName(args, false);
@@ -298,20 +387,39 @@ public class PurgeCommand extends Consumer {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_NOTICE_1));
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_NOTICE_2));
while (!Consumer.pausedSuccess) {
while (!Consumer.pausedSuccess && !Consumer.isPersistenceHalted()) {
Thread.sleep(1);
}
if (Consumer.isPersistenceHalted()) {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.DATABASE_BUSY));
return;
}
Consumer.isPaused = true;
consumerPaused = true;
requirePurgeNotCancelled();
if (ConfigHandler.databaseType.isClickHouse()) {
connection.close();
connection = null;
if (optimize) {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_OPTIMIZING));
}
requirePurgeNotCancelled();
removed = Database.purgeClickHouse(timeStart, timeEnd, argWid, includeBlockIdsFinal, optimize);
EntitySpawnTracking.invalidateDatabaseVerification();
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_SUCCESS));
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_ROWS, NumberFormat.getInstance().format(removed), (removed == 1 ? Selector.FIRST : Selector.SECOND)));
return;
}
String query = "";
PreparedStatement preparedStmt = null;
boolean abort = false;
String purgePrefix = "tmp_" + ConfigHandler.prefix;
if (!Config.getGlobal().MYSQL) {
if (ConfigHandler.databaseType.isSQLite()) {
query = "ATTACH DATABASE '" + ConfigHandler.path + ConfigHandler.sqlite + ".tmp' AS tmp_db";
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
purgePrefix = "tmp_db." + ConfigHandler.prefix;
@@ -324,32 +432,40 @@ public class PurgeCommand extends Consumer {
return;
}
if (!Config.getGlobal().MYSQL) {
if (ConfigHandler.databaseType.isDuckDB()) {
transactionStatement = createPurgeStatement(connection);
Database.beginTransaction(transactionStatement, ConfigHandler.databaseType);
duckPurgeStarted = true;
duckTransaction = true;
}
if (ConfigHandler.databaseType.isSQLite()) {
for (String table : ConfigHandler.databaseTables) {
requirePurgeNotCancelled();
try {
query = "DROP TABLE IF EXISTS " + purgePrefix + table + "";
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
}
catch (Exception e) {
ErrorReporter.report(e);
reportPurgeFailure(e);
}
}
Database.createDatabaseTables(purgePrefix, false, null, Config.getGlobal().MYSQL, true);
Database.createDatabaseTables(purgePrefix, false, null, ConfigHandler.databaseType, true);
}
List<String> worldTables = Arrays.asList("sign", "container", "entity_container", "entity_interaction", "item", "session", "chat", "command", "block");
List<String> restrictTables = Arrays.asList("block");
List<String> excludeTables = Arrays.asList("database_lock"); // don't insert data into these tables
for (String table : ConfigHandler.databaseTables) {
requirePurgeNotCancelled();
String tableName = table.replaceAll("_", " ");
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_PROCESSING, tableName));
if (!Config.getGlobal().MYSQL) {
if (ConfigHandler.databaseType.isSQLite()) {
String columns = "";
ResultSet rs = connection.createStatement().executeQuery("SELECT * FROM " + purgePrefix + table);
Statement columnStatement = createPurgeStatement(connection);
ResultSet rs = columnStatement.executeQuery("SELECT * FROM " + purgePrefix + table);
ResultSetMetaData resultSetMetaData = rs.getMetaData();
int columnCount = resultSetMetaData.getColumnCount();
for (int i = 1; i <= columnCount; i++) {
@@ -362,6 +478,7 @@ public class PurgeCommand extends Consumer {
}
}
rs.close();
columnStatement.close();
String insertColumns = "";
String selectColumns = columns;
if (table.equals("block")) {
@@ -377,16 +494,16 @@ public class PurgeCommand extends Consumer {
if (table.equals("entity_spawn")) {
timeLimit = " WHERE removed=0 OR block_rowid IN(SELECT rowid FROM " + purgePrefix + "block) OR kill_rowid IN(SELECT rowid FROM " + purgePrefix + "entity) OR rowid IN(SELECT entity_spawn_rowid FROM " + purgePrefix + "entity_container) OR rowid IN(SELECT entity_spawn_rowid FROM " + purgePrefix + "entity_interaction)";
}
else if (PURGE_TABLES.contains(table)) {
else if (PurgePolicy.isPurgeable(table)) {
String blockRestriction = "(";
if (hasBlockRestriction && restrictTables.contains(table)) {
if (hasBlockRestriction && PurgePolicy.supportsBlockRestriction(table)) {
blockRestriction = "action IN(" + LookupActions.ENTITY_KILL + "," + LookupActions.ENTITY_SPAWN + ") OR type NOT IN(" + includeBlockFinal + ") OR (type IN(" + includeBlockFinal + ") AND ";
}
else if (hasBlockRestriction) {
purge = false;
}
if (argWid > 0 && worldTables.contains(table)) {
if (argWid > 0 && PurgePolicy.isWorldScoped(table)) {
if (table.equals("entity_container") || table.equals("entity_interaction")) {
if (purge) {
String worldMatch = "(wid = '" + argWid + "' OR entity_spawn_rowid IN(SELECT rowid FROM " + ConfigHandler.prefix + "entity_spawn WHERE current_wid = '" + argWid + "'))";
@@ -402,58 +519,59 @@ public class PurgeCommand extends Consumer {
}
}
query = "INSERT INTO " + purgePrefix + table + insertColumns + " SELECT " + selectColumns + " FROM " + ConfigHandler.prefix + table + timeLimit;
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
}
catch (Exception e) {
error = true;
ErrorReporter.report(e);
reportPurgeFailure(e);
}
}
if (error) {
requirePurgeNotCancelled();
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_ERROR, tableName));
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_REPAIRING));
try {
query = "DELETE FROM " + purgePrefix + table;
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
}
catch (Exception e) {
ErrorReporter.report(e);
reportPurgeFailure(e);
}
try {
query = "REINDEX " + ConfigHandler.prefix + table;
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
}
catch (Exception e) {
ErrorReporter.report(e);
reportPurgeFailure(e);
}
try {
String index = " NOT INDEXED";
query = "INSERT INTO " + purgePrefix + table + insertColumns + " SELECT " + selectColumns + " FROM " + ConfigHandler.prefix + table + index;
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
}
catch (Exception e) {
ErrorReporter.report(e);
reportPurgeFailure(e);
abort = true;
break;
}
try {
boolean purge = PURGE_TABLES.contains(table);
boolean purge = PurgePolicy.isPurgeable(table);
String blockRestriction = "";
if (hasBlockRestriction && restrictTables.contains(table)) {
if (hasBlockRestriction && PurgePolicy.supportsBlockRestriction(table)) {
blockRestriction = "action NOT IN(" + LookupActions.ENTITY_KILL + "," + LookupActions.ENTITY_SPAWN + ") AND type IN(" + includeBlockFinal + ") AND ";
}
else if (hasBlockRestriction) {
@@ -461,7 +579,7 @@ public class PurgeCommand extends Consumer {
}
String worldRestriction = "";
if (argWid > 0 && worldTables.contains(table)) {
if (argWid > 0 && PurgePolicy.isWorldScoped(table)) {
if (table.equals("entity_container") || table.equals("entity_interaction")) {
worldRestriction = " AND (wid = '" + argWid + "' OR entity_spawn_rowid IN(SELECT rowid FROM " + ConfigHandler.prefix + "entity_spawn WHERE current_wid = '" + argWid + "'))";
}
@@ -475,21 +593,21 @@ public class PurgeCommand extends Consumer {
if (purge) {
query = "DELETE FROM " + purgePrefix + table + " WHERE " + blockRestriction + "time < '" + timeEnd + "' AND time >= '" + timeStart + "'" + worldRestriction;
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
}
}
catch (Exception e) {
ErrorReporter.report(e);
reportPurgeFailure(e);
}
}
if (PURGE_TABLES.contains(table) || table.equals("entity_spawn")) {
if (PurgePolicy.isPurgeable(table) || table.equals("entity_spawn")) {
int oldCount = 0;
try {
query = "SELECT COUNT(*) as count FROM " + ConfigHandler.prefix + table + " LIMIT 0, 1";
preparedStmt = connection.prepareStatement(query);
query = "SELECT COUNT(*) as count FROM " + ConfigHandler.prefix + table + " LIMIT 1 OFFSET 0";
preparedStmt = preparePurgeStatement(connection, query);
ResultSet resultSet = preparedStmt.executeQuery();
while (resultSet.next()) {
oldCount = resultSet.getInt("count");
@@ -498,13 +616,13 @@ public class PurgeCommand extends Consumer {
preparedStmt.close();
}
catch (Exception e) {
ErrorReporter.report(e);
reportPurgeFailure(e);
}
int new_count = 0;
try {
query = "SELECT COUNT(*) as count FROM " + purgePrefix + table + " LIMIT 0, 1";
preparedStmt = connection.prepareStatement(query);
query = "SELECT COUNT(*) as count FROM " + purgePrefix + table + " LIMIT 1 OFFSET 0";
preparedStmt = preparePurgeStatement(connection, query);
ResultSet resultSet = preparedStmt.executeQuery();
while (resultSet.next()) {
new_count = resultSet.getInt("count");
@@ -513,19 +631,19 @@ public class PurgeCommand extends Consumer {
preparedStmt.close();
}
catch (Exception e) {
ErrorReporter.report(e);
reportPurgeFailure(e);
}
removed = removed + (oldCount - new_count);
}
}
if (Config.getGlobal().MYSQL) {
if (!ConfigHandler.databaseType.isSQLite()) {
try {
boolean purge = PURGE_TABLES.contains(table);
boolean purge = PurgePolicy.isPurgeable(table);
String blockRestriction = "";
if (hasBlockRestriction && restrictTables.contains(table)) {
if (hasBlockRestriction && PurgePolicy.supportsBlockRestriction(table)) {
blockRestriction = "action NOT IN(" + LookupActions.ENTITY_KILL + "," + LookupActions.ENTITY_SPAWN + ") AND type IN(" + includeBlockFinal + ") AND ";
}
else if (hasBlockRestriction) {
@@ -533,7 +651,7 @@ public class PurgeCommand extends Consumer {
}
String worldRestriction = "";
if (argWid > 0 && worldTables.contains(table)) {
if (argWid > 0 && PurgePolicy.isWorldScoped(table)) {
if (table.equals("entity_container") || table.equals("entity_interaction")) {
worldRestriction = " AND (wid = '" + argWid + "' OR entity_spawn_rowid IN(SELECT rowid FROM " + ConfigHandler.prefix + "entity_spawn WHERE current_wid = '" + argWid + "'))";
}
@@ -547,95 +665,184 @@ public class PurgeCommand extends Consumer {
if (purge) {
query = "DELETE FROM " + ConfigHandler.prefix + table + " WHERE " + blockRestriction + "time < '" + timeEnd + "' AND time >= '" + timeStart + "'" + worldRestriction;
preparedStmt = connection.prepareStatement(query);
preparedStmt.execute();
removed = removed + preparedStmt.getUpdateCount();
preparedStmt = preparePurgeStatement(connection, query);
removed = removed + preparedStmt.executeUpdate();
preparedStmt.close();
}
}
catch (Exception e) {
if (ConfigHandler.databaseType.isDuckDB()) {
throw e;
}
if (!ConfigHandler.serverRunning) {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_FAILED));
return;
}
ErrorReporter.report(e);
reportPurgeFailure(e);
}
}
}
String retainedPrefix = Config.getGlobal().MYSQL ? ConfigHandler.prefix : purgePrefix;
requirePurgeNotCancelled();
String retainedPrefix = ConfigHandler.databaseType.isSQLite() ? purgePrefix : ConfigHandler.prefix;
query = "UPDATE " + retainedPrefix + "entity_spawn SET kill_rowid=NULL WHERE kill_rowid IS NOT NULL AND NOT EXISTS (SELECT 1 FROM " + retainedPrefix + "entity WHERE " + retainedPrefix + "entity.rowid=" + retainedPrefix + "entity_spawn.kill_rowid)";
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.executeUpdate();
preparedStmt.close();
requirePurgeNotCancelled();
query = "UPDATE " + retainedPrefix + "entity_spawn SET block_rowid=NULL WHERE block_rowid IS NOT NULL AND NOT EXISTS (SELECT 1 FROM " + retainedPrefix + "block WHERE " + retainedPrefix + "block.rowid=" + retainedPrefix + "entity_spawn.block_rowid)";
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.executeUpdate();
preparedStmt.close();
requirePurgeNotCancelled();
query = "DELETE FROM " + retainedPrefix + "entity_interaction WHERE NOT EXISTS (SELECT 1 FROM " + retainedPrefix + "entity_spawn WHERE " + retainedPrefix + "entity_spawn.rowid=" + retainedPrefix + "entity_interaction.entity_spawn_rowid)";
preparedStmt = connection.prepareStatement(query);
preparedStmt.executeUpdate();
removed = removed + preparedStmt.getUpdateCount();
preparedStmt = preparePurgeStatement(connection, query);
removed = removed + preparedStmt.executeUpdate();
preparedStmt.close();
requirePurgeNotCancelled();
query = "DELETE FROM " + retainedPrefix + "entity_container WHERE NOT EXISTS (SELECT 1 FROM " + retainedPrefix + "entity_spawn WHERE " + retainedPrefix + "entity_spawn.rowid=" + retainedPrefix + "entity_container.entity_spawn_rowid)";
preparedStmt = connection.prepareStatement(query);
preparedStmt.executeUpdate();
removed = removed + preparedStmt.getUpdateCount();
preparedStmt = preparePurgeStatement(connection, query);
removed = removed + preparedStmt.executeUpdate();
preparedStmt.close();
requirePurgeNotCancelled();
query = "DELETE FROM " + retainedPrefix + "entity_spawn WHERE removed=1 AND block_rowid IS NULL AND kill_rowid IS NULL AND NOT EXISTS (SELECT 1 FROM " + retainedPrefix + "entity_container WHERE " + retainedPrefix + "entity_container.entity_spawn_rowid=" + retainedPrefix + "entity_spawn.rowid) AND NOT EXISTS (SELECT 1 FROM " + retainedPrefix + "entity_interaction WHERE " + retainedPrefix + "entity_interaction.entity_spawn_rowid=" + retainedPrefix + "entity_spawn.rowid)";
preparedStmt = connection.prepareStatement(query);
preparedStmt.executeUpdate();
removed = removed + preparedStmt.getUpdateCount();
preparedStmt = preparePurgeStatement(connection, query);
removed = removed + preparedStmt.executeUpdate();
preparedStmt.close();
if (Config.getGlobal().MYSQL && optimize) {
if (ConfigHandler.databaseType.isMySQL() && optimize) {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_OPTIMIZING));
for (String table : ConfigHandler.databaseTables) {
requirePurgeNotCancelled();
query = "OPTIMIZE LOCAL TABLE " + ConfigHandler.prefix + table + "";
preparedStmt = connection.prepareStatement(query);
preparedStmt = preparePurgeStatement(connection, query);
preparedStmt.execute();
preparedStmt.close();
}
}
if (duckTransaction) {
activePurgeStatement = transactionStatement;
requirePurgeNotCancelled();
if (!Database.commitTransactionChecked(transactionStatement, ConfigHandler.databaseType)) {
throw new SQLException("Unable to commit DuckDB purge transaction");
}
duckTransaction = false;
duckPurgeStarted = false;
requirePurgeNotCancelled();
try {
transactionStatement.execute("CHECKPOINT");
}
catch (SQLException e) {
if (!shutdownCancellationRequested) {
ErrorReporter.report(e);
}
}
transactionStatement.close();
transactionStatement = null;
}
connection.close();
connection = null;
requirePurgeNotCancelled();
try {
Consumer.lockDatabaseMaintenanceInterruptibly();
maintenanceLocked = true;
if (!Database.awaitConnectionDrain(CONNECTION_DRAIN_TIMEOUT_MILLIS)) {
throw new SQLException("Timed out waiting for active database connections before purge handoff");
}
requirePurgeNotCancelled();
}
catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw exception;
}
if (abort) {
if (!Config.getGlobal().MYSQL) {
(new File(ConfigHandler.path + ConfigHandler.sqlite + ".tmp")).delete();
if (ConfigHandler.databaseType.isSQLite()) {
Files.deleteIfExists(sqliteTempDatabase());
}
resumePersistence = false;
handoffStarted = true;
ConfigHandler.loadDatabase();
handoffStarted = false;
resumePersistence = true;
Chat.sendGlobalMessage(player, Color.RED + Phrase.build(Phrase.PURGE_ABORTED));
return;
}
if (!Config.getGlobal().MYSQL) {
(new File(ConfigHandler.path + ConfigHandler.sqlite)).delete();
(new File(ConfigHandler.path + ConfigHandler.sqlite + ".tmp")).renameTo(new File(ConfigHandler.path + ConfigHandler.sqlite));
resumePersistence = false;
handoffStarted = true;
if (ConfigHandler.databaseType.isSQLite()) {
replaceSQLiteDatabase();
}
ConfigHandler.loadDatabase();
handoffStarted = false;
resumePersistence = true;
EntitySpawnTracking.invalidateDatabaseVerification();
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_SUCCESS));
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_ROWS, NumberFormat.getInstance().format(removed), (removed == 1 ? Selector.FIRST : Selector.SECOND)));
}
catch (Exception e) {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_FAILED));
ErrorReporter.report(e);
boolean shutdownCancelled = shutdownCancellationRequested || e instanceof InterruptedException;
if (duckTransaction && transactionStatement != null) {
duckRollbackSucceeded = Database.rollbackTransaction(transactionStatement, ConfigHandler.databaseType);
}
if (ConfigHandler.databaseType.isDuckDB() && duckPurgeStarted && !duckRollbackSucceeded) {
Consumer.haltPersistence();
}
if (handoffStarted) {
Consumer.requireDatabaseReload();
}
if (shutdownCancelled) {
Thread.currentThread().interrupt();
}
else {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.PURGE_FAILED));
if (handoffStarted) {
Chat.sendGlobalMessage(player, Phrase.build(Phrase.RELOAD_FAILED));
}
ErrorReporter.report(e);
}
}
finally {
if (consumerPaused) {
if (transactionStatement != null) {
try {
transactionStatement.close();
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
if (connection != null) {
try {
connection.close();
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
if (maintenanceLocked) {
Consumer.unlockDatabaseMaintenance();
}
if (consumerPaused && resumePersistence && !Consumer.isPersistenceHalted()) {
Consumer.isPaused = false;
}
if (purgeClaimed) {
Consumer.releasePurge();
}
if (activePurgeThread == Thread.currentThread()) {
activePurgeStatement = null;
activePurgeThread = null;
}
}
}
}
@@ -644,4 +851,14 @@ public class PurgeCommand extends Consumer {
Thread thread = new Thread(runnable);
thread.start();
}
private static Path sqliteTempDatabase() {
return Path.of(ConfigHandler.path + ConfigHandler.sqlite + ".tmp");
}
private static void replaceSQLiteDatabase() throws IOException {
Path database = Path.of(ConfigHandler.path + ConfigHandler.sqlite);
Path temporary = sqliteTempDatabase();
Files.move(temporary, database, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
}
}
src/main/java/net/coreprotect/command/ReloadCommand.java
+52 -16
View File
@@ -4,13 +4,18 @@ import org.bukkit.command.CommandSender;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.database.Database;
import net.coreprotect.language.Phrase;
import net.coreprotect.thread.NetworkHandler;
import net.coreprotect.utility.Chat;
import net.coreprotect.utility.Color;
import net.coreprotect.utility.EntitySpawnTracking;
import net.coreprotect.utility.ErrorReporter;
public class ReloadCommand {
private static final long CONNECTION_DRAIN_TIMEOUT_MILLIS = 60_000L;
protected static void runCommand(final CommandSender player, boolean permission, String[] args) {
if (permission) {
if (ConfigHandler.converterRunning) {
@@ -34,26 +39,14 @@ public class ReloadCommand {
@Override
public void run() {
try {
if (Consumer.isPaused) {
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.RELOAD_STARTED));
}
while (Consumer.isPaused) {
Thread.sleep(1);
}
Consumer.isPaused = true;
ConfigHandler.performInitialization(false);
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.RELOAD_SUCCESS));
Thread networkHandler = new Thread(new NetworkHandler(false, false));
networkHandler.start();
performReload(player);
}
catch (Exception e) {
ErrorReporter.report(e);
}
Consumer.isPaused = false;
ConfigHandler.lookupThrottle.put(player.getName(), new Object[] { false, System.currentTimeMillis() });
finally {
ConfigHandler.lookupThrottle.put(player.getName(), new Object[] { false, System.currentTimeMillis() });
}
}
}
Runnable runnable = new BasicThread();
@@ -64,4 +57,47 @@ public class ReloadCommand {
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.NO_PERMISSION));
}
}
static synchronized boolean performReload(CommandSender player) throws InterruptedException {
boolean resumePersistence = !Consumer.isDatabaseReloadPaused();
Consumer.OperationStartResult startResult = Consumer.beginDatabaseReload();
if (startResult != Consumer.OperationStartResult.STARTED) {
Phrase phrase = startResult == Consumer.OperationStartResult.PURGE_RUNNING ? Phrase.PURGE_IN_PROGRESS
: startResult == Consumer.OperationStartResult.ROLLBACK_RUNNING ? Phrase.ROLLBACK_IN_PROGRESS
: startResult == Consumer.OperationStartResult.PERSISTENCE_HALTED ? Phrase.DATABASE_PERSISTENCE_HALTED : Phrase.DATABASE_BUSY;
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(phrase));
return false;
}
long deadline = System.nanoTime() + CONNECTION_DRAIN_TIMEOUT_MILLIS * 1_000_000L;
try {
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.RELOAD_STARTED));
if (!Consumer.lockDatabaseReload(CONNECTION_DRAIN_TIMEOUT_MILLIS)) {
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.RELOAD_FAILED));
return false;
}
if (!Database.awaitConnectionDrain(remainingMillis(deadline))) {
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.RELOAD_FAILED));
return false;
}
resumePersistence = false;
if (!ConfigHandler.performInitialization(false)) {
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.RELOAD_FAILED));
return false;
}
EntitySpawnTracking.invalidateDatabaseVerification();
resumePersistence = true;
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(Phrase.RELOAD_SUCCESS));
Thread networkHandler = new Thread(new NetworkHandler(false, false));
networkHandler.start();
return true;
}
finally {
Consumer.endDatabaseReload(resumePersistence);
}
}
private static long remainingMillis(long deadline) {
return Math.max(0L, (deadline - System.nanoTime()) / 1_000_000L);
}
}
src/main/java/net/coreprotect/command/RollbackRestoreCommand.java
+5 -3
View File
@@ -355,9 +355,11 @@ public class RollbackRestoreCommand {
final String[] finalArgs = args;
final int finalPreview = preview;
Consumer.OperationStartResult startResult = Consumer.claimRollback(player.getName());
Consumer.OperationStartResult startResult = Consumer.claimRollback(player.getName(), finalPreview == 0);
if (startResult != Consumer.OperationStartResult.STARTED) {
Phrase phrase = startResult == Consumer.OperationStartResult.PURGE_RUNNING ? Phrase.PURGE_IN_PROGRESS : Phrase.ROLLBACK_IN_PROGRESS;
Phrase phrase = startResult == Consumer.OperationStartResult.PURGE_RUNNING ? Phrase.PURGE_IN_PROGRESS
: startResult == Consumer.OperationStartResult.PERSISTENCE_HALTED ? Phrase.DATABASE_PERSISTENCE_HALTED
: startResult == Consumer.OperationStartResult.RELOAD_RUNNING ? Phrase.DATABASE_BUSY : Phrase.ROLLBACK_IN_PROGRESS;
Chat.sendMessage(player, Color.DARK_AQUA + "CoreProtect " + Color.WHITE + "- " + Phrase.build(phrase));
return;
}
@@ -445,7 +447,7 @@ public class RollbackRestoreCommand {
boolean completed = true;
if (finalArgAction.contains(5) && ConfigHandler.lookupEntityContainer.get(player2.getName()) == null) {
ContainerRollback.performContainerRollbackRestore(statement, player2, uuidList, rollbackusers2, rtime, blist, elist, euserlist, finalArgAction, location, radius, finalTimeStart, finalTimeEnd, restrictWorld, false, verbose, action);
completed = ContainerRollback.performContainerRollbackRestore(statement, player2, uuidList, rollbackusers2, rtime, blist, elist, euserlist, finalArgAction, location, radius, finalTimeStart, finalTimeEnd, restrictWorld, false, verbose, action);
}
else {
completed = Rollback.performRollbackRestore(statement, player2, uuidList, rollbackusers2, rtime, blist, elist, euserlist, finalArgAction, finalEntityActionFilter, location, radius, finalTimeStart, finalTimeEnd, restrictWorld, false, verbose, action, finalPreview) != null;
src/main/java/net/coreprotect/command/StatusCommand.java
+7 -5
View File
@@ -67,16 +67,18 @@ public class StatusCommand {
// Using MySQL/SQLite (Database Size: 587MB)
String firstVersion = Patch.getFirstVersion();
String firstVersion = ConfigHandler.databaseType.isClickHouse() ? "" : Patch.getFirstVersion();
if (firstVersion.length() > 0) {
firstVersion = " (" + Phrase.build(Phrase.FIRST_VERSION, firstVersion) + ")";
}
if (Config.getGlobal().MYSQL) {
Chat.sendMessage(player, Color.DARK_AQUA + Phrase.build(Phrase.STATUS_DATABASE, Color.WHITE, "MySQL") + firstVersion);
String databaseName = ConfigHandler.databaseType.getDisplayName();
if (Consumer.isPersistenceHalted()) {
databaseName += " " + Color.RED + Phrase.build(Phrase.STATUS_DATABASE_STATE, Selector.FIRST) + Color.WHITE;
}
else {
Chat.sendMessage(player, Color.DARK_AQUA + Phrase.build(Phrase.STATUS_DATABASE, Color.WHITE, "SQLite") + firstVersion);
else if (!ConfigHandler.databaseReachable) {
databaseName += " " + Color.RED + Phrase.build(Phrase.STATUS_DATABASE_STATE, Selector.SECOND) + Color.WHITE;
}
Chat.sendMessage(player, Color.DARK_AQUA + Phrase.build(Phrase.STATUS_DATABASE, Color.WHITE, databaseName) + firstVersion);
if (ConfigHandler.worldeditEnabled) {
Chat.sendMessage(player, Color.DARK_AQUA + Phrase.build(Phrase.STATUS_INTEGRATION, Color.WHITE, "WorldEdit", Selector.FIRST));
src/main/java/net/coreprotect/command/lookup/StandardLookupThread.java
+5 -8
View File
@@ -166,8 +166,8 @@ public class StandardLookupThread implements Runnable {
break;
}
else if (actions.contains(LookupActions.USERNAME)) {
if (ConfigHandler.uuidCache.get(check.toLowerCase(Locale.ROOT)) != null) {
String uuid = ConfigHandler.uuidCache.get(check.toLowerCase(Locale.ROOT));
String uuid = UserStatement.getUuid(connection, check);
if (uuid != null) {
uuidList.add(uuid);
}
}
@@ -366,7 +366,7 @@ public class StandardLookupThread implements Runnable {
else if (actions.contains(LookupActions.USERNAME)) {
for (String[] data : lookupList) {
String time = data[0];
String user = ConfigHandler.uuidCacheReversed.get(data[1]);
String user = UserStatement.getNameByUuid(data[1]);
String username = data[2];
String timeago = ChatUtils.getTimeSince(Integer.parseInt(time), unixtimestamp, true);
Chat.sendComponent(player, timeago + " " + Color.WHITE + "- " + Phrase.build(Phrase.LOOKUP_USERNAME, Color.DARK_AQUA + user + Color.WHITE, Color.DARK_AQUA + username + Color.WHITE));
@@ -503,10 +503,7 @@ public class StandardLookupThread implements Runnable {
}
else if ((daction == LookupActions.ENTITY_KILL || daction == LookupActions.ENTITY_SPAWN) && !actions.contains(LookupActions.ITEM) && amount == -1) {
if (daction == LookupActions.ENTITY_KILL && dtype == 0) {
if (ConfigHandler.playerIdCacheReversed.get(ddata) == null) {
UserStatement.loadName(connection, ddata);
}
dname = ConfigHandler.playerIdCacheReversed.get(ddata);
dname = UserStatement.getName(connection, ddata);
isPlayer = true;
}
else {
@@ -651,7 +648,7 @@ public class StandardLookupThread implements Runnable {
String rowsFound = Phrase.build(Phrase.LOOKUP_ROWS_FOUND, numberFormat.format(recordRows), recordRows == 1 ? Selector.FIRST : Selector.SECOND);
Chat.sendMessage(player, Color.WHITE + "----- " + Color.DARK_AQUA + "CoreProtect" + Color.WHITE + " | " + Color.DARK_AQUA + rowsFound + Color.WHITE + " -----");
for (LookupSummaryRow row : summaryRows) {
String userName = UserStatement.loadName(connection, row.getUserId());
String userName = UserStatement.getName(connection, row.getUserId());
if (userName == null || userName.isEmpty()) {
userName = "unknown";
}
src/main/java/net/coreprotect/command/parser/LocationParser.java
+12 -8
View File
@@ -4,6 +4,7 @@ import java.util.Locale;
import org.bukkit.Bukkit;
import org.bukkit.Location;
import org.bukkit.World;
import org.bukkit.command.BlockCommandSender;
import org.bukkit.command.CommandSender;
import org.bukkit.entity.Player;
@@ -95,18 +96,21 @@ public class LocationParser {
}
}
if (cCount > 1) {
if (location == null && worldId > 0) {
if (worldId > 0) {
location = new Location(Bukkit.getWorld(WorldUtils.getWorldName(worldId)), 0, 0, 0);
}
if (location != null) {
int worldMaxHeight = location.getWorld().getMaxHeight() - 1;
int worldMinHeight = BukkitAdapter.ADAPTER.getMinHeight(location.getWorld());
World world = location.getWorld();
if (world != null) {
int worldMaxHeight = world.getMaxHeight() - 1;
int worldMinHeight = BukkitAdapter.ADAPTER.getMinHeight(world);
if (y < worldMinHeight) {
y = Double.valueOf(worldMinHeight);
}
if (y > worldMaxHeight) {
y = Double.valueOf(worldMaxHeight);
if (y < worldMinHeight) {
y = Double.valueOf(worldMinHeight);
}
if (y > worldMaxHeight) {
y = Double.valueOf(worldMaxHeight);
}
}
location.setX(x);
src/main/java/net/coreprotect/command/parser/WorldParser.java
+2 -1
View File
@@ -45,7 +45,8 @@ public class WorldParser {
else if (inputProcessed.startsWith("#")) {
world_id = WorldUtils.matchWorld(inputProcessed);
if (world_id == -1 && !requireLoaded) {
world_id = ConfigHandler.worlds.getOrDefault(argument.replaceFirst("#", ""), -1);
String worldName = argument.replaceFirst("#", "");
world_id = ConfigHandler.worlds.getOrDefault(worldName, -1);
}
}
next = 0;
src/main/java/net/coreprotect/config/Config.java
+43 -3
View File
@@ -35,6 +35,13 @@ public class Config extends Language {
private Config defaults;
public String DONATION_KEY;
public String DATABASE_TYPE;
public String CLICKHOUSE_HOST;
public String CLICKHOUSE_DATABASE;
public String CLICKHOUSE_USERNAME;
public String CLICKHOUSE_PASSWORD;
public String DUCKDB_MEMORY_LIMIT;
public String DUCKDB_MAX_TEMP_DIRECTORY_SIZE;
public String PREFIX;
public String MYSQL_HOST;
public String MYSQL_DATABASE;
@@ -44,6 +51,7 @@ public class Config extends Language {
public String AUTO_PURGE;
public String AUTO_PURGE_TIME;
public boolean ENABLE_SSL;
public boolean CLICKHOUSE_TLS;
public boolean DISABLE_WAL;
public boolean HOVER_EVENTS;
public boolean DATABASE_LOCK;
@@ -95,19 +103,30 @@ public class Config extends Language {
public boolean USERNAME_CHANGES;
public boolean WORLDEDIT;
public int MAXIMUM_POOL_SIZE;
public int CLICKHOUSE_PORT;
public int MYSQL_PORT;
public int DEFAULT_RADIUS;
public int DUCKDB_THREADS;
public int MAX_RADIUS;
static {
DEFAULT_VALUES.put("donation-key", "");
DEFAULT_VALUES.put("use-mysql", "false");
DEFAULT_VALUES.put("database-type", "duckdb");
DEFAULT_VALUES.put("table-prefix", "co_");
DEFAULT_VALUES.put("mysql-host", "127.0.0.1");
DEFAULT_VALUES.put("mysql-port", "3306");
DEFAULT_VALUES.put("mysql-database", "database");
DEFAULT_VALUES.put("mysql-username", "root");
DEFAULT_VALUES.put("mysql-password", "");
DEFAULT_VALUES.put("clickhouse-host", "127.0.0.1");
DEFAULT_VALUES.put("clickhouse-port", "8123");
DEFAULT_VALUES.put("clickhouse-database", "default");
DEFAULT_VALUES.put("clickhouse-username", "default");
DEFAULT_VALUES.put("clickhouse-password", "");
DEFAULT_VALUES.put("clickhouse-tls", "false");
DEFAULT_VALUES.put("duckdb-memory-limit", "512MB");
DEFAULT_VALUES.put("duckdb-threads", "2");
DEFAULT_VALUES.put("duckdb-max-temp-directory-size", "10GB");
DEFAULT_VALUES.put("language", "en");
DEFAULT_VALUES.put("auto-purge", "false");
DEFAULT_VALUES.put("check-updates", "true");
@@ -154,7 +173,10 @@ public class Config extends Language {
DEFAULT_VALUES.put("worldedit", "true");
HEADERS.put("donation-key", new String[] { "# CoreProtect is donationware. Obtain a donation key from coreprotect.net/donate/" });
HEADERS.put("use-mysql", new String[] { "# MySQL is optional and not required.", "# If you prefer to use MySQL, enable the following and fill out the fields." });
HEADERS.put("database-type", new String[] { "# Database engine used by CoreProtect. Valid values are duckdb, clickhouse, sqlite, and mysql.", "# Run /co reload or restart the server after changing the database engine or connection target." });
HEADERS.put("mysql-host", new String[] { "# Connection settings for MySQL." });
HEADERS.put("clickhouse-host", new String[] { "# Connection settings for ClickHouse 25.6 or newer.", "# The configured database must already exist; CoreProtect creates its prefixed tables and views.", "# ClickHouse currently supports one CoreProtect installation per database and table prefix.", "# Preserve .clickhouse-writer with this installation and do not copy it to another active server." });
HEADERS.put("duckdb-memory-limit", new String[] { "# Resource limits for the embedded DuckDB database.", "# The memory limit controls DuckDB's buffer manager; the temporary limit caps spill data and is not preallocated." });
HEADERS.put("language", new String[] { "# If modified, will automatically attempt to translate languages phrases.", "# List of language codes: https://coreprotect.net/languages/" });
HEADERS.put("auto-purge", new String[] { "# Automatically purge data older than the configured time.", "# Examples: 30d, 12w, 6mo. Set to false to disable." });
HEADERS.put("check-updates", new String[] { "# If enabled, CoreProtect will check for updates when your server starts up.", "# If an update is available, you'll be notified via your server console.", });
@@ -214,6 +236,13 @@ public class Config extends Language {
this.UNKNOWN_LOGGING = this.getBoolean("unknown-logging", false);
this.MAXIMUM_POOL_SIZE = this.getInt("maximum-pool-size", 10);
this.DONATION_KEY = this.getString("donation-key");
this.DATABASE_TYPE = this.getString("database-type");
this.CLICKHOUSE_HOST = this.getString("clickhouse-host");
this.CLICKHOUSE_PORT = this.getInt("clickhouse-port");
this.CLICKHOUSE_DATABASE = this.getString("clickhouse-database");
this.CLICKHOUSE_USERNAME = this.getString("clickhouse-username");
this.CLICKHOUSE_PASSWORD = this.getString("clickhouse-password");
this.CLICKHOUSE_TLS = this.getBoolean("clickhouse-tls");
this.MYSQL = this.getBoolean("use-mysql");
this.PREFIX = this.getString("table-prefix");
this.MYSQL_HOST = this.getString("mysql-host");
@@ -221,6 +250,9 @@ public class Config extends Language {
this.MYSQL_DATABASE = this.getString("mysql-database");
this.MYSQL_USERNAME = this.getString("mysql-username");
this.MYSQL_PASSWORD = this.getString("mysql-password");
this.DUCKDB_MEMORY_LIMIT = this.getString("duckdb-memory-limit");
this.DUCKDB_THREADS = this.getInt("duckdb-threads", 2);
this.DUCKDB_MAX_TEMP_DIRECTORY_SIZE = this.getString("duckdb-max-temp-directory-size");
this.LANGUAGE = this.getString("language");
this.AUTO_PURGE = this.getString("auto-purge");
this.AUTO_PURGE_TIME = this.getString("auto-purge-time");
@@ -350,6 +382,10 @@ public class Config extends Language {
this.config.clear();
}
public boolean hasOption(String key) {
return this.config.containsKey(key);
}
public void loadDefaults() {
this.clearConfig();
this.readValues();
@@ -390,6 +426,7 @@ public class Config extends Language {
final Config temp = new Config();
temp.loadDefaults();
temp.addMissingOptions(globalFile);
map.put("config", Files.readAllBytes(globalFile.toPath()));
}
for (final File worldConfigFile : configFolder.listFiles((File file) -> file.getName().endsWith(".yml"))) {
@@ -477,7 +514,7 @@ public class Config extends Language {
for (final Map.Entry<String, String> entry : DEFAULT_VALUES.entrySet()) {
final String key = entry.getKey();
final String defaultValue = entry.getValue();
String defaultValue = entry.getValue();
final String configuredValue = this.config.get(key);
@@ -487,6 +524,9 @@ public class Config extends Language {
if (key.equals("auto-purge") && VersionUtils.isCommunityEdition()) {
continue;
}
if (key.equals("database-type") && !writeHeader) {
defaultValue = this.getBoolean("use-mysql") ? "mysql" : "sqlite";
}
final String[] header = HEADERS.get(key);
src/main/java/net/coreprotect/config/ConfigHandler.java
+270 -177
View File
@@ -4,6 +4,7 @@ import java.io.File;
import java.io.RandomAccessFile;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Collections;
@@ -28,7 +29,11 @@ import com.zaxxer.hikari.HikariDataSource;
import net.coreprotect.bukkit.BukkitAdapter;
import net.coreprotect.consumer.Queue;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.Database;
import net.coreprotect.database.DatabaseType;
import net.coreprotect.database.DuckDBNativeSupport;
import net.coreprotect.database.clickhouse.ClickHouseJdbcConfig;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.language.Phrase;
import net.coreprotect.listener.ListenerHandler;
@@ -62,6 +67,10 @@ public class ConfigHandler extends Queue {
public static final String LATEST_VERSION = "26.2";
public static String path = "plugins/CoreProtect/";
public static String sqlite = "database.db";
public static String duckdb = "database.duckdb";
public static String duckdbMemoryLimit = "512MB";
public static String duckdbMaxTempDirectorySize = "10GB";
public static int duckdbThreads = 2;
public static String host = "127.0.0.1";
public static int port = 3306;
public static String database = "database";
@@ -70,6 +79,7 @@ public class ConfigHandler extends Queue {
public static String prefix = "co_";
public static String prefixConfig = "co_";
public static int maximumPoolSize = 10;
public static DatabaseType databaseType = DatabaseType.SQLITE;
public static final String BLACKLIST_COMMENT_SEPARATOR = ";";
public static final String BLACKLIST_FILTER_SEPARATOR = "@";
@@ -81,6 +91,7 @@ public class ConfigHandler extends Queue {
public static final boolean isPaper = VersionUtils.isPaper();
public static final boolean isFolia = VersionUtils.isFolia();
public static volatile boolean serverRunning = false;
public static volatile boolean shutdownDrainRunning = false;
public static volatile boolean converterRunning = false;
public static volatile boolean purgeRunning = false;
public static volatile boolean migrationRunning = false;
@@ -93,21 +104,55 @@ public class ConfigHandler extends Queue {
public static volatile int entityId = 0;
public static volatile int artId = 0;
public static final AtomicLong autoPurgeRowsPurged = new AtomicLong(0);
private static boolean duckDBFallbackAllowed = false;
private static <K, V> Map<K, V> syncMap() {
return Collections.synchronizedMap(new HashMap<>());
}
public static Map<String, Integer> worlds = syncMap();
public static Map<Integer, String> worldsReversed = syncMap();
public static Map<String, Integer> materials = syncMap();
public static Map<Integer, String> materialsReversed = syncMap();
public static Map<String, Integer> blockdata = syncMap();
public static Map<Integer, String> blockdataReversed = syncMap();
public static Map<String, Integer> entities = syncMap();
public static Map<Integer, String> entitiesReversed = syncMap();
public static Map<String, Integer> art = syncMap();
public static Map<Integer, String> artReversed = syncMap();
private static <K, V> Map<K, V> syncMap(Map<K, V> values) {
return Collections.synchronizedMap(new HashMap<>(values));
}
private static IdentifierCache loadIdentifierCache(Statement statement, String table, String valueColumn) throws SQLException {
Map<String, Integer> values = new HashMap<>();
Map<Integer, String> reversed = new HashMap<>();
int maximumId = 0;
try (ResultSet resultSet = statement.executeQuery("SELECT id," + valueColumn + " FROM " + ConfigHandler.prefix + table)) {
while (resultSet.next()) {
int id = resultSet.getInt("id");
String value = resultSet.getString(valueColumn);
values.put(value, id);
reversed.put(id, value);
maximumId = Math.max(maximumId, id);
}
}
return new IdentifierCache(values, reversed, maximumId);
}
private static final class IdentifierCache {
private final Map<String, Integer> values;
private final Map<Integer, String> reversed;
private int maximumId;
private IdentifierCache(Map<String, Integer> values, Map<Integer, String> reversed, int maximumId) {
this.values = values;
this.reversed = reversed;
this.maximumId = maximumId;
}
}
public static volatile Map<String, Integer> worlds = syncMap();
public static volatile Map<Integer, String> worldsReversed = syncMap();
public static volatile Map<String, Integer> materials = syncMap();
public static volatile Map<Integer, String> materialsReversed = syncMap();
public static volatile Map<String, Integer> blockdata = syncMap();
public static volatile Map<Integer, String> blockdataReversed = syncMap();
public static volatile Map<String, Integer> entities = syncMap();
public static volatile Map<Integer, String> entitiesReversed = syncMap();
public static volatile Map<String, Integer> art = syncMap();
public static volatile Map<Integer, String> artReversed = syncMap();
public static Map<String, int[]> rollbackHash = syncMap();
public static Map<String, Boolean> inspecting = syncMap();
public static Map<String, Boolean> blacklist = syncMap();
@@ -163,6 +208,24 @@ public class ConfigHandler extends Queue {
public static void checkPlayers(Connection connection) {
ConfigHandler.playerIdCache.clear();
ConfigHandler.playerIdCacheReversed.clear();
if (ConfigHandler.databaseType.isClickHouse()) {
if (Bukkit.getServer().getOnlinePlayers().isEmpty()) {
return;
}
try (ConsumerWriteBatch batch = Database.openConsumerWriteBatch(connection)) {
batch.begin();
for (Player player : Bukkit.getServer().getOnlinePlayers()) {
batch.resolveUserId(player.getName(), player.getUniqueId().toString());
}
if (!batch.commit()) {
throw new IllegalStateException("Unable to initialize ClickHouse player records");
}
}
catch (Exception exception) {
throw new IllegalStateException("Unable to initialize ClickHouse player records", exception);
}
return;
}
for (Player player : Bukkit.getServer().getOnlinePlayers()) {
if (ConfigHandler.playerIdCache.get(player.getName().toLowerCase(Locale.ROOT)) == null) {
UserStatement.loadId(connection, player.getName(), player.getUniqueId().toString());
@@ -240,38 +303,78 @@ public class ConfigHandler extends Queue {
}
}
private static void loadConfig() {
try {
Config.init();
ConfigFile.init(ConfigFile.LANGUAGE); // load user phrases
ConfigFile.init(ConfigFile.LANGUAGE_CACHE); // load translation cache
private static void loadConfig() throws Exception {
File configFolder = new File(ConfigHandler.path);
File configFile = new File(configFolder, ConfigFile.CONFIG);
File[] existingFiles = configFolder.listFiles();
duckDBFallbackAllowed = !configFile.exists() && (!configFolder.exists() || (existingFiles != null && existingFiles.length == 0));
// Enforce "co_" table prefix if using SQLite.
if (!Config.getGlobal().MYSQL) {
ConfigHandler.prefixConfig = Config.getGlobal().PREFIX;
Config.getGlobal().PREFIX = "co_";
Config.init();
ConfigFile.init(ConfigFile.LANGUAGE); // load user phrases
ConfigFile.init(ConfigFile.LANGUAGE_CACHE); // load translation cache
Config global = Config.getGlobal();
if (global.hasOption("database-type")) {
ConfigHandler.databaseType = DatabaseType.parse(global.DATABASE_TYPE, global.MYSQL);
if (global.hasOption("use-mysql") && global.MYSQL != ConfigHandler.databaseType.isMySQL()) {
Chat.sendConsoleMessage(Color.YELLOW + "[CoreProtect] " + Phrase.build(Phrase.DATABASE_TYPE_OVERRIDE));
}
ConfigHandler.host = Config.getGlobal().MYSQL_HOST;
ConfigHandler.port = Config.getGlobal().MYSQL_PORT;
ConfigHandler.database = Config.getGlobal().MYSQL_DATABASE;
ConfigHandler.username = Config.getGlobal().MYSQL_USERNAME;
ConfigHandler.password = Config.getGlobal().MYSQL_PASSWORD;
ConfigHandler.maximumPoolSize = Config.getGlobal().MAXIMUM_POOL_SIZE;
ConfigHandler.prefix = Config.getGlobal().PREFIX;
ConfigHandler.loadBlacklist(); // Load the blacklist file if it exists.
}
catch (Exception e) {
ErrorReporter.report(e);
else {
ConfigHandler.databaseType = global.MYSQL ? DatabaseType.MYSQL : DatabaseType.SQLITE;
}
global.MYSQL = ConfigHandler.databaseType.isMySQL();
ConfigHandler.prefixConfig = global.PREFIX;
// Embedded databases use the fixed prefix expected by maintenance operations.
if (ConfigHandler.databaseType.isEmbedded()) {
global.PREFIX = "co_";
}
ConfigHandler.host = global.MYSQL_HOST;
ConfigHandler.port = global.MYSQL_PORT;
ConfigHandler.database = global.MYSQL_DATABASE;
ConfigHandler.username = global.MYSQL_USERNAME;
ConfigHandler.password = global.MYSQL_PASSWORD;
ConfigHandler.maximumPoolSize = global.MAXIMUM_POOL_SIZE;
ConfigHandler.duckdbMemoryLimit = global.DUCKDB_MEMORY_LIMIT;
ConfigHandler.duckdbThreads = Math.max(1, global.DUCKDB_THREADS);
ConfigHandler.duckdbMaxTempDirectorySize = global.DUCKDB_MAX_TEMP_DIRECTORY_SIZE;
ConfigHandler.prefix = global.PREFIX;
ConfigHandler.loadBlacklist(); // Load the blacklist file if it exists.
}
public static void loadDatabase() {
// close old pool when we reload the database, e.g. in purge command
ConfigHandler.databaseReachable = false;
Database.closeConnection();
try {
initializeDatabase();
ConfigHandler.databaseReachable = true;
}
catch (RuntimeException exception) {
Database.closeConnection();
throw exception;
}
}
if (!Config.getGlobal().MYSQL) {
private static void initializeDatabase() {
if (ConfigHandler.databaseType.isClickHouse()) {
try {
if (!Config.getGlobal().DATABASE_LOCK) {
throw new IllegalStateException("ClickHouse requires database-lock to remain enabled");
}
Config global = Config.getGlobal();
ClickHouseJdbcConfig config = new ClickHouseJdbcConfig(global.CLICKHOUSE_HOST, global.CLICKHOUSE_PORT, global.CLICKHOUSE_DATABASE, global.CLICKHOUSE_USERNAME, global.CLICKHOUSE_PASSWORD, global.CLICKHOUSE_TLS);
Database.initializeClickHouse(config, ConfigHandler.prefix);
}
catch (Exception exception) {
throw new IllegalStateException("Failed to initialize ClickHouse", exception);
}
return;
}
if (ConfigHandler.databaseType.isEmbedded()) {
try {
File tempFile = File.createTempFile("CoreProtect_" + System.currentTimeMillis(), ".tmp");
tempFile.setExecutable(true);
@@ -295,7 +398,27 @@ public class ConfigHandler extends Queue {
tempFile.delete();
Class.forName("org.sqlite.JDBC");
if (ConfigHandler.databaseType.isDuckDB() && duckDBFallbackAllowed) {
try {
DuckDBNativeSupport.verifyAvailable();
}
catch (Throwable failure) {
if (!DuckDBNativeSupport.isNativeUnavailable(failure)) {
throw new IllegalStateException("Unable to verify DuckDB on this system", failure);
}
DatabaseConfigWriter.persistDatabaseType(DatabaseType.SQLITE);
ConfigHandler.databaseType = DatabaseType.SQLITE;
Config global = Config.getGlobal();
global.DATABASE_TYPE = DatabaseType.SQLITE.name().toLowerCase(Locale.ROOT);
global.MYSQL = false;
Chat.sendConsoleMessage(Color.YELLOW + "[CoreProtect] " + Phrase.build(Phrase.DATABASE_FALLBACK, DatabaseType.DUCKDB.getDisplayName(), DatabaseType.SQLITE.getDisplayName()));
}
finally {
duckDBFallbackAllowed = false;
}
}
Class.forName(ConfigHandler.databaseType.isDuckDB() ? "org.duckdb.DuckDBDriver" : "org.sqlite.JDBC");
}
catch (Exception e) {
ErrorReporter.report(e);
@@ -315,9 +438,12 @@ public class ConfigHandler extends Queue {
config.setUsername(ConfigHandler.username);
config.setPassword(ConfigHandler.password);
config.setMaximumPoolSize(ConfigHandler.maximumPoolSize);
config.setConnectionTimeout(10000);
// Keep connection age below short limits used by some managed MySQL hosts.
config.setMaxLifetime(60000);
config.setKeepaliveTime(0);
config.addDataSourceProperty("characterEncoding", "UTF-8");
config.addDataSourceProperty("connectionTimeout", "10000");
config.addDataSourceProperty("connectTimeout", "10000");
/* https://github.com/brettwooldridge/HikariCP/wiki/MySQL-Configuration */
/* https://cdn.oreillystatic.com/en/assets/1/event/21/Connector_J%20Performance%20Gems%20Presentation.pdf */
config.addDataSourceProperty("cachePrepStmts", "true");
@@ -335,114 +461,75 @@ public class ConfigHandler extends Queue {
ConfigHandler.hikariDataSource = new HikariDataSource(config);
}
Database.createDatabaseTables(ConfigHandler.prefix, false, null, Config.getGlobal().MYSQL, false);
Database.createDatabaseTables(ConfigHandler.prefix, false, null, ConfigHandler.databaseType, false);
}
public static void loadMaterials(Statement statement) {
public static boolean loadMaterials(Statement statement) {
try {
String query = "SELECT id,material FROM " + ConfigHandler.prefix + "material_map";
ResultSet rs = statement.executeQuery(query);
ConfigHandler.materials.clear();
ConfigHandler.materialsReversed.clear();
materialId = 0;
while (rs.next()) {
int id = rs.getInt("id");
String material = rs.getString("material");
ConfigHandler.materials.put(material, id);
ConfigHandler.materialsReversed.put(id, material);
if (id > materialId) {
materialId = id;
}
}
rs.close();
IdentifierCache cache = loadIdentifierCache(statement, "material_map", "material");
ConfigHandler.materials = syncMap(cache.values);
ConfigHandler.materialsReversed = syncMap(cache.reversed);
materialId = cache.maximumId;
return true;
}
catch (Exception e) {
ErrorReporter.report(e);
return false;
}
}
public static void loadBlockdata(Statement statement) {
public static boolean loadBlockdata(Statement statement) {
try {
String query = "SELECT id,data FROM " + ConfigHandler.prefix + "blockdata_map";
ResultSet rs = statement.executeQuery(query);
ConfigHandler.blockdata.clear();
ConfigHandler.blockdataReversed.clear();
blockdataId = 0;
while (rs.next()) {
int id = rs.getInt("id");
String data = rs.getString("data");
ConfigHandler.blockdata.put(data, id);
ConfigHandler.blockdataReversed.put(id, data);
if (id > blockdataId) {
blockdataId = id;
}
}
rs.close();
IdentifierCache cache = loadIdentifierCache(statement, "blockdata_map", "data");
ConfigHandler.blockdata = syncMap(cache.values);
ConfigHandler.blockdataReversed = syncMap(cache.reversed);
blockdataId = cache.maximumId;
return true;
}
catch (Exception e) {
ErrorReporter.report(e);
return false;
}
}
public static void loadArt(Statement statement) {
public static boolean loadArt(Statement statement) {
try {
String query = "SELECT id,art FROM " + ConfigHandler.prefix + "art_map";
ResultSet rs = statement.executeQuery(query);
ConfigHandler.art.clear();
ConfigHandler.artReversed.clear();
artId = 0;
while (rs.next()) {
int id = rs.getInt("id");
String art = rs.getString("art");
ConfigHandler.art.put(art, id);
ConfigHandler.artReversed.put(id, art);
if (id > artId) {
artId = id;
}
}
rs.close();
IdentifierCache cache = loadIdentifierCache(statement, "art_map", "art");
ConfigHandler.art = syncMap(cache.values);
ConfigHandler.artReversed = syncMap(cache.reversed);
artId = cache.maximumId;
return true;
}
catch (Exception e) {
ErrorReporter.report(e);
return false;
}
}
public static void loadEntities(Statement statement) {
public static boolean loadEntities(Statement statement) {
try {
String query = "SELECT id,entity FROM " + ConfigHandler.prefix + "entity_map";
ResultSet rs = statement.executeQuery(query);
ConfigHandler.entities.clear();
ConfigHandler.entitiesReversed.clear();
entityId = 0;
while (rs.next()) {
int id = rs.getInt("id");
String entity = rs.getString("entity");
ConfigHandler.entities.put(entity, id);
ConfigHandler.entitiesReversed.put(id, entity);
if (id > entityId) {
entityId = id;
}
}
rs.close();
IdentifierCache cache = loadIdentifierCache(statement, "entity_map", "entity");
ConfigHandler.entities = syncMap(cache.values);
ConfigHandler.entitiesReversed = syncMap(cache.reversed);
entityId = cache.maximumId;
return true;
}
catch (Exception e) {
ErrorReporter.report(e);
return false;
}
}
public static void loadTypes(Statement statement) {
loadMaterials(statement);
loadBlockdata(statement);
loadArt(statement);
loadEntities(statement);
public static boolean loadTypes(Statement statement) {
boolean loaded = loadMaterials(statement);
loaded &= loadBlockdata(statement);
loaded &= loadArt(statement);
loaded &= loadEntities(statement);
return loaded;
}
/**
* Unified method to reload cache from database when DATABASE_LOCK is false (multi-server setup)
* Reloads an identifier cache when multi-server cache refresh is permitted.
*
* @param type
* The type of cache to reload
@@ -450,84 +537,84 @@ public class ConfigHandler extends Queue {
* The name to look up after reload
* @return The ID if found after reload, or -1 if not found
*/
public static int reloadAndGetId(CacheType type, String name) {
// Only reload if DATABASE_LOCK is false (multi-server setup)
public static synchronized int reloadAndGetId(CacheType type, String name) {
if (Config.getGlobal().DATABASE_LOCK) {
return -1;
}
try (Connection connection = Database.getConnection(true)) {
if (connection != null) {
Statement statement = connection.createStatement();
// Reload appropriate cache based on type
switch (type) {
case MATERIALS:
loadMaterials(statement);
statement.close();
return materials.getOrDefault(name, -1);
case BLOCKDATA:
loadBlockdata(statement);
statement.close();
return blockdata.getOrDefault(name, -1);
case ART:
loadArt(statement);
statement.close();
return art.getOrDefault(name, -1);
case ENTITIES:
loadEntities(statement);
statement.close();
return entities.getOrDefault(name, -1);
case WORLDS:
loadWorlds(statement);
statement.close();
return worlds.getOrDefault(name, -1);
default:
statement.close();
return -1;
}
}
}
catch (Exception e) {
ErrorReporter.report(e);
if (!reloadIdentifierCache(type)) {
return -1;
}
return -1;
switch (type) {
case MATERIALS:
return materials.getOrDefault(name, -1);
case BLOCKDATA:
return blockdata.getOrDefault(name, -1);
case ART:
return art.getOrDefault(name, -1);
case ENTITIES:
return entities.getOrDefault(name, -1);
case WORLDS:
return worlds.getOrDefault(name, -1);
default:
return -1;
}
}
public static void loadWorlds(Statement statement) {
try {
String query = "SELECT id,world FROM " + ConfigHandler.prefix + "world";
ResultSet rs = statement.executeQuery(query);
ConfigHandler.worlds.clear();
ConfigHandler.worldsReversed.clear();
worldId = 0;
while (rs.next()) {
int id = rs.getInt("id");
String world = rs.getString("world");
ConfigHandler.worlds.put(world, id);
ConfigHandler.worldsReversed.put(id, world);
if (id > worldId) {
worldId = id;
}
private static boolean reloadIdentifierCache(CacheType type) {
try (Connection connection = Database.getConnection(true)) {
if (connection == null) {
return false;
}
rs.close();
List<World> worlds = Bukkit.getServer().getWorlds();
for (World world : worlds) {
String worldname = world.getName();
if (ConfigHandler.worlds.get(worldname) == null) {
int id = worldId + 1;
ConfigHandler.worlds.put(worldname, id);
ConfigHandler.worldsReversed.put(id, worldname);
worldId = id;
Queue.queueWorldInsert(id, worldname);
try (Statement statement = connection.createStatement()) {
switch (type) {
case MATERIALS:
return loadMaterials(statement);
case BLOCKDATA:
return loadBlockdata(statement);
case ART:
return loadArt(statement);
case ENTITIES:
return loadEntities(statement);
case WORLDS:
return loadWorlds(statement);
default:
return false;
}
}
}
catch (Exception e) {
ErrorReporter.report(e);
return false;
}
}
public static boolean loadWorlds(Statement statement) {
try {
IdentifierCache cache = loadIdentifierCache(statement, "world", "world");
List<World> worlds = Bukkit.getServer().getWorlds();
Map<Integer, String> queuedWorlds = new HashMap<>();
for (World world : worlds) {
String worldname = world.getName();
if (!cache.values.containsKey(worldname)) {
int id = ++cache.maximumId;
cache.values.put(worldname, id);
cache.reversed.put(id, worldname);
queuedWorlds.put(id, worldname);
}
}
ConfigHandler.worlds = syncMap(cache.values);
ConfigHandler.worldsReversed = syncMap(cache.reversed);
worldId = cache.maximumId;
for (Map.Entry<Integer, String> world : queuedWorlds.entrySet()) {
Queue.queueWorldInsert(world.getKey(), world.getValue());
}
return true;
}
catch (Exception e) {
ErrorReporter.report(e);
return false;
}
}
@@ -588,14 +675,18 @@ public class ConfigHandler extends Queue {
}
catch (Exception e) {
ErrorReporter.report(e);
return false;
}
try (Connection connection = Database.getConnection(true, 0)) {
Statement statement = connection.createStatement();
ConfigHandler.checkPlayers(connection);
ConfigHandler.loadWorlds(statement); // Load world ID's into memory.
ConfigHandler.loadTypes(statement); // Load material ID's into memory.
boolean worldsLoaded = ConfigHandler.loadWorlds(statement); // Load world ID's into memory.
boolean typesLoaded = ConfigHandler.loadTypes(statement); // Load material ID's into memory.
if (ConfigHandler.databaseType.isColumnar() && (!worldsLoaded || !typesLoaded)) {
throw new IllegalStateException("Unable to initialize columnar database identifier caches");
}
// Initialize WorldEdit logging
if (VersionUtils.checkWorldEdit()) {
@@ -609,7 +700,9 @@ public class ConfigHandler extends Queue {
VersionUtils.unloadWorldEdit();
}
ConfigHandler.serverRunning = true; // Set as running before patching
if (startup) {
ConfigHandler.serverRunning = true; // Set as running before patching
}
boolean validVersion = Patch.versionCheck(statement); // Minor upgrades & version check
boolean databaseLock = true;
if (startup) {
src/main/java/net/coreprotect/config/DatabaseConfigWriter.java
+237
View File
@@ -0,0 +1,237 @@
package net.coreprotect.config;
import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.AtomicMoveNotSupportedException;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.nio.file.attribute.AclFileAttributeView;
import java.nio.file.attribute.DosFileAttributeView;
import java.nio.file.attribute.DosFileAttributes;
import java.nio.file.attribute.FileOwnerAttributeView;
import java.nio.file.attribute.PosixFileAttributeView;
import java.nio.file.attribute.PosixFileAttributes;
import java.util.Arrays;
import java.util.Locale;
import net.coreprotect.database.DatabaseType;
public final class DatabaseConfigWriter {
private DatabaseConfigWriter() {
throw new IllegalStateException("Utility class");
}
public static synchronized void persistDatabaseType(DatabaseType type) throws IOException {
Path configFile = Paths.get(ConfigHandler.path).resolve(ConfigFile.CONFIG);
if (!Files.isRegularFile(configFile, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("CoreProtect config is not a regular file: " + configFile);
}
byte[] original = Files.readAllBytes(configFile);
byte[] updated = updateDatabaseType(original, type);
if (Arrays.equals(original, updated)) {
return;
}
Path directory = configFile.toAbsolutePath().getParent();
if (directory == null) {
throw new IOException("CoreProtect config has no parent directory: " + configFile);
}
Path temporary = Files.createTempFile(directory, ".config.yml.", ".tmp");
boolean moved = false;
try {
Files.write(temporary, updated, StandardOpenOption.TRUNCATE_EXISTING);
try (FileChannel channel = FileChannel.open(temporary, StandardOpenOption.WRITE)) {
channel.force(true);
}
copyAttributes(configFile, temporary);
try {
Files.move(temporary, configFile, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
}
catch (AtomicMoveNotSupportedException exception) {
throw new IOException("The CoreProtect config filesystem does not support atomic replacement", exception);
}
moved = true;
forceDirectory(directory);
}
finally {
if (!moved) {
Files.deleteIfExists(temporary);
}
}
}
private static byte[] updateDatabaseType(byte[] data, DatabaseType type) {
String content = new String(data, StandardCharsets.UTF_8);
String databaseType = type.name().toLowerCase(Locale.ROOT);
String lineSeparator = detectLineSeparator(content);
boolean terminalLineSeparator = content.endsWith("\n") || content.endsWith("\r");
boolean foundDatabaseType = false;
StringBuilder updated = new StringBuilder(content.length() + 32);
int offset = 0;
while (offset < content.length()) {
int lineEnd = offset;
while (lineEnd < content.length() && content.charAt(lineEnd) != '\r' && content.charAt(lineEnd) != '\n') {
lineEnd++;
}
int nextOffset = lineEnd;
if (nextOffset < content.length()) {
char separator = content.charAt(nextOffset++);
if (separator == '\r' && nextOffset < content.length() && content.charAt(nextOffset) == '\n') {
nextOffset++;
}
}
String line = content.substring(offset, lineEnd);
String replaced = replaceScalar(line, "database-type", databaseType);
if (replaced != null) {
foundDatabaseType = true;
updated.append(replaced);
}
else {
replaced = replaceScalar(line, "use-mysql", Boolean.toString(type.isMySQL()));
updated.append(replaced == null ? line : replaced);
}
updated.append(content, lineEnd, nextOffset);
offset = nextOffset;
}
if (!foundDatabaseType) {
if (updated.length() > 0 && updated.charAt(updated.length() - 1) != '\n' && updated.charAt(updated.length() - 1) != '\r') {
updated.append(lineSeparator);
}
updated.append("database-type: ").append(databaseType);
if (terminalLineSeparator) {
updated.append(lineSeparator);
}
}
return updated.toString().getBytes(StandardCharsets.UTF_8);
}
private static String replaceScalar(String line, String expectedKey, String value) {
int first = 0;
while (first < line.length() && Character.isWhitespace(line.charAt(first))) {
first++;
}
if (first == line.length() || line.charAt(first) == '#') {
return null;
}
int colon = line.indexOf(':', first);
if (colon < 0 || !line.substring(first, colon).trim().equalsIgnoreCase(expectedKey)) {
return null;
}
int valueStart = colon + 1;
while (valueStart < line.length() && Character.isWhitespace(line.charAt(valueStart))) {
valueStart++;
}
int comment = findComment(line, valueStart);
int valueEnd = comment < 0 ? line.length() : comment;
while (valueEnd > valueStart && Character.isWhitespace(line.charAt(valueEnd - 1))) {
valueEnd--;
}
String replacement = value;
if (valueStart < valueEnd && (line.charAt(valueStart) == '\'' || line.charAt(valueStart) == '"')) {
char quote = line.charAt(valueStart);
replacement = quote + value + quote;
}
String whitespace = line.substring(colon + 1, valueStart);
if (whitespace.isEmpty()) {
whitespace = " ";
}
String suffix = line.substring(valueEnd);
if (comment == valueStart && suffix.startsWith("#")) {
suffix = " " + suffix;
}
return line.substring(0, colon + 1) + whitespace + replacement + suffix;
}
private static int findComment(String line, int start) {
boolean singleQuoted = false;
boolean doubleQuoted = false;
boolean escaped = false;
for (int index = start; index < line.length(); index++) {
char character = line.charAt(index);
if (doubleQuoted && character == '\\' && !escaped) {
escaped = true;
continue;
}
if (character == '"' && !singleQuoted && !escaped) {
doubleQuoted = !doubleQuoted;
}
else if (character == '\'' && !doubleQuoted) {
singleQuoted = !singleQuoted;
}
else if (character == '#' && !singleQuoted && !doubleQuoted && (index == start || Character.isWhitespace(line.charAt(index - 1)))) {
return index;
}
escaped = false;
}
return -1;
}
private static String detectLineSeparator(String content) {
int newline = content.indexOf('\n');
if (newline >= 0) {
return newline > 0 && content.charAt(newline - 1) == '\r' ? "\r\n" : "\n";
}
return content.indexOf('\r') >= 0 ? "\r" : System.lineSeparator();
}
private static void copyAttributes(Path source, Path target) throws IOException {
boolean ownerCopied = false;
PosixFileAttributeView sourcePosix = Files.getFileAttributeView(source, PosixFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
PosixFileAttributeView targetPosix = Files.getFileAttributeView(target, PosixFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
if (sourcePosix != null && targetPosix != null) {
PosixFileAttributes attributes = sourcePosix.readAttributes();
targetPosix.setOwner(attributes.owner());
targetPosix.setGroup(attributes.group());
targetPosix.setPermissions(attributes.permissions());
ownerCopied = true;
}
AclFileAttributeView sourceAcl = Files.getFileAttributeView(source, AclFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
AclFileAttributeView targetAcl = Files.getFileAttributeView(target, AclFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
if (sourceAcl != null && targetAcl != null) {
targetAcl.setOwner(sourceAcl.getOwner());
targetAcl.setAcl(sourceAcl.getAcl());
ownerCopied = true;
}
if (!ownerCopied) {
FileOwnerAttributeView sourceOwner = Files.getFileAttributeView(source, FileOwnerAttributeView.class, LinkOption.NOFOLLOW_LINKS);
FileOwnerAttributeView targetOwner = Files.getFileAttributeView(target, FileOwnerAttributeView.class, LinkOption.NOFOLLOW_LINKS);
if (sourceOwner != null && targetOwner != null) {
targetOwner.setOwner(sourceOwner.getOwner());
}
}
DosFileAttributeView sourceDos = Files.getFileAttributeView(source, DosFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
DosFileAttributeView targetDos = Files.getFileAttributeView(target, DosFileAttributeView.class, LinkOption.NOFOLLOW_LINKS);
if (sourceDos != null && targetDos != null) {
DosFileAttributes attributes = sourceDos.readAttributes();
targetDos.setArchive(attributes.isArchive());
targetDos.setHidden(attributes.isHidden());
targetDos.setSystem(attributes.isSystem());
targetDos.setReadOnly(attributes.isReadOnly());
}
}
private static void forceDirectory(Path directory) {
try (FileChannel channel = FileChannel.open(directory, StandardOpenOption.READ)) {
channel.force(true);
}
catch (IOException | UnsupportedOperationException ignored) {
}
}
}
src/main/java/net/coreprotect/consumer/Consumer.java
+239 -5
View File
@@ -6,6 +6,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import org.bukkit.Bukkit;
import org.bukkit.block.BlockState;
@@ -14,6 +15,9 @@ import org.bukkit.inventory.ItemStack;
import net.coreprotect.CoreProtect;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.process.Process;
import net.coreprotect.language.Phrase;
import net.coreprotect.utility.Chat;
import net.coreprotect.utility.Color;
import net.coreprotect.utility.ErrorReporter;
public class Consumer extends Process implements Runnable, Thread.UncaughtExceptionHandler {
@@ -21,17 +25,26 @@ public class Consumer extends Process implements Runnable, Thread.UncaughtExcept
public enum OperationStartResult {
STARTED,
PURGE_RUNNING,
ROLLBACK_RUNNING
ROLLBACK_RUNNING,
RELOAD_RUNNING,
PERSISTENCE_HALTED,
INTERRUPTED
}
private static Thread consumerThread = null;
private static final ReentrantReadWriteLock databaseLifecycle = new ReentrantReadWriteLock(true);
private static final Object rollbackPurgeGate = new Object();
private static long pendingRollbackPublications = 0;
private static volatile boolean backgroundPurgeRunning = false;
private static volatile boolean backgroundPurgePausesPersistence = false;
private static volatile boolean databaseReloadPaused = false;
private static volatile boolean databaseReloadRunning = false;
public static volatile int currentConsumer = 0;
public static volatile boolean isPaused = false;
public static volatile boolean transacting = false;
public static volatile boolean interrupt = false;
protected static volatile boolean pausedSuccess = false;
private static volatile boolean persistenceHalted = false;
public static ConcurrentHashMap<Integer, ArrayList<Object[]>> consumer = new ConcurrentHashMap<>(4, 0.75f, 2);
// public static ConcurrentHashMap<Integer, Integer[]> consumer_id = new ConcurrentHashMap<>();
@@ -98,6 +111,14 @@ public class Consumer extends Process implements Runnable, Thread.UncaughtExcept
}
public static void initialize() {
synchronized (rollbackPurgeGate) {
persistenceHalted = false;
pendingRollbackPublications = 0;
}
databaseReloadPaused = false;
databaseReloadRunning = false;
backgroundPurgeRunning = false;
backgroundPurgePausesPersistence = false;
Consumer.consumer.put(0, new ArrayList<>());
Consumer.consumer.put(1, new ArrayList<>());
Consumer.consumer_id.put(0, new Integer[] { 0, 0 });
@@ -126,9 +147,120 @@ public class Consumer extends Process implements Runnable, Thread.UncaughtExcept
return consumerThread != null && consumerThread.isAlive();
}
public static boolean isPersistenceHalted() {
return persistenceHalted;
}
public static void lockDatabaseAccess() {
databaseLifecycle.readLock().lock();
}
public static void unlockDatabaseAccess() {
databaseLifecycle.readLock().unlock();
}
public static void lockDatabaseMaintenance() {
databaseLifecycle.writeLock().lock();
}
public static void lockDatabaseMaintenanceInterruptibly() throws InterruptedException {
databaseLifecycle.writeLock().lockInterruptibly();
}
public static void unlockDatabaseMaintenance() {
databaseLifecycle.writeLock().unlock();
}
public static boolean isDatabaseReloadBlocked() {
return databaseReloadPaused && !databaseLifecycle.isWriteLockedByCurrentThread();
}
public static boolean isDatabaseReloadPaused() {
return databaseReloadPaused;
}
public static boolean isDatabaseReloadRunning() {
return databaseReloadRunning;
}
public static boolean isBackgroundPurgeRunning() {
return backgroundPurgeRunning;
}
public static void requireDatabaseReload() {
synchronized (rollbackPurgeGate) {
databaseReloadPaused = true;
}
}
public static OperationStartResult beginDatabaseReload() {
synchronized (rollbackPurgeGate) {
if (persistenceHalted) {
return OperationStartResult.PERSISTENCE_HALTED;
}
if (databaseReloadRunning) {
return OperationStartResult.RELOAD_RUNNING;
}
if (ConfigHandler.purgeRunning || backgroundPurgeRunning) {
return OperationStartResult.PURGE_RUNNING;
}
if (!ConfigHandler.activeRollbacks.isEmpty() || pendingRollbackPublications > 0) {
return OperationStartResult.ROLLBACK_RUNNING;
}
databaseReloadRunning = true;
databaseReloadPaused = true;
}
return OperationStartResult.STARTED;
}
public static void lockDatabaseReload() {
databaseLifecycle.writeLock().lock();
}
public static boolean lockDatabaseReload(long timeoutMillis) throws InterruptedException {
return databaseLifecycle.writeLock().tryLock(Math.max(0L, timeoutMillis), java.util.concurrent.TimeUnit.MILLISECONDS);
}
public static void endDatabaseReload(boolean resumePersistence) {
try {
synchronized (rollbackPurgeGate) {
databaseReloadRunning = false;
if (resumePersistence) {
databaseReloadPaused = false;
if (!ConfigHandler.converterRunning) {
isPaused = false;
}
}
}
}
finally {
if (databaseLifecycle.isWriteLockedByCurrentThread()) {
databaseLifecycle.writeLock().unlock();
}
}
}
public static synchronized void haltPersistence() {
synchronized (rollbackPurgeGate) {
if (persistenceHalted) {
return;
}
persistenceHalted = true;
}
ConfigHandler.databaseReachable = false;
Chat.sendConsoleMessage(Color.RED + "[CoreProtect] " + Phrase.build(Phrase.DATABASE_PERSISTENCE_HALTED));
Chat.sendConsoleMessage(Color.RED + "[CoreProtect] " + Phrase.build(Phrase.DATABASE_QUEUE_LOSS));
}
public static OperationStartResult claimPurge() {
synchronized (rollbackPurgeGate) {
if (ConfigHandler.purgeRunning) {
if (persistenceHalted) {
return OperationStartResult.PERSISTENCE_HALTED;
}
if (databaseReloadPaused) {
return OperationStartResult.RELOAD_RUNNING;
}
if (ConfigHandler.purgeRunning || backgroundPurgeRunning) {
return OperationStartResult.PURGE_RUNNING;
}
if (!ConfigHandler.activeRollbacks.isEmpty() || pendingRollbackPublications > 0) {
@@ -145,9 +277,97 @@ public class Consumer extends Process implements Runnable, Thread.UncaughtExcept
}
}
public static OperationStartResult claimRollback(String user) {
public static OperationStartResult claimBackgroundPurge() {
boolean pausePersistence;
synchronized (rollbackPurgeGate) {
if (ConfigHandler.purgeRunning) {
pausePersistence = ConfigHandler.databaseType.isClickHouse();
OperationStartResult result = beginBackgroundPurge(pausePersistence);
if (result != OperationStartResult.STARTED) {
return result;
}
}
return lockBackgroundPurge(pausePersistence);
}
public static OperationStartResult claimBackgroundPurge(boolean pausePersistence) {
synchronized (rollbackPurgeGate) {
OperationStartResult result = beginBackgroundPurge(pausePersistence);
if (result != OperationStartResult.STARTED) {
return result;
}
}
return lockBackgroundPurge(pausePersistence);
}
private static OperationStartResult beginBackgroundPurge(boolean pausePersistence) {
if (persistenceHalted) {
return OperationStartResult.PERSISTENCE_HALTED;
}
if (databaseReloadPaused) {
return OperationStartResult.RELOAD_RUNNING;
}
if (ConfigHandler.purgeRunning || backgroundPurgeRunning) {
return OperationStartResult.PURGE_RUNNING;
}
if (!ConfigHandler.activeRollbacks.isEmpty() || pendingRollbackPublications > 0) {
return OperationStartResult.ROLLBACK_RUNNING;
}
backgroundPurgeRunning = true;
backgroundPurgePausesPersistence = pausePersistence;
return OperationStartResult.STARTED;
}
private static OperationStartResult lockBackgroundPurge(boolean pausePersistence) {
if (pausePersistence) {
try {
databaseLifecycle.writeLock().lockInterruptibly();
}
catch (InterruptedException exception) {
synchronized (rollbackPurgeGate) {
backgroundPurgePausesPersistence = false;
backgroundPurgeRunning = false;
}
Thread.currentThread().interrupt();
return OperationStartResult.INTERRUPTED;
}
synchronized (rollbackPurgeGate) {
if (persistenceHalted) {
databaseLifecycle.writeLock().unlock();
backgroundPurgePausesPersistence = false;
backgroundPurgeRunning = false;
return OperationStartResult.PERSISTENCE_HALTED;
}
}
}
return OperationStartResult.STARTED;
}
public static void releaseBackgroundPurge() {
synchronized (rollbackPurgeGate) {
if (!backgroundPurgeRunning) {
return;
}
if (backgroundPurgePausesPersistence) {
databaseLifecycle.writeLock().unlock();
}
backgroundPurgePausesPersistence = false;
backgroundPurgeRunning = false;
}
}
public static OperationStartResult claimRollback(String user) {
return claimRollback(user, true);
}
public static OperationStartResult claimRollback(String user, boolean requiresPersistence) {
synchronized (rollbackPurgeGate) {
if (requiresPersistence && persistenceHalted) {
return OperationStartResult.PERSISTENCE_HALTED;
}
if (databaseReloadPaused) {
return OperationStartResult.RELOAD_RUNNING;
}
if (ConfigHandler.purgeRunning || backgroundPurgeRunning) {
return OperationStartResult.PURGE_RUNNING;
}
if (ConfigHandler.activeRollbacks.containsKey(user)) {
@@ -206,6 +426,12 @@ public class Consumer extends Process implements Runnable, Thread.UncaughtExcept
if (!ConfigHandler.serverRunning && !ConfigHandler.converterRunning) {
lastRun = true;
}
if (persistenceHalted) {
if (!lastRun) {
errorDelay();
}
continue;
}
try {
int process_id = 0;
synchronized (Consumer.consumer_id) {
@@ -219,7 +445,15 @@ public class Consumer extends Process implements Runnable, Thread.UncaughtExcept
}
Thread.sleep(500);
pauseConsumer(process_id);
Process.processConsumer(process_id, lastRun);
databaseLifecycle.readLock().lock();
try {
if (!databaseReloadPaused && !isPaused && !persistenceHalted) {
Process.processConsumer(process_id, lastRun);
}
}
finally {
databaseLifecycle.readLock().unlock();
}
}
catch (Exception e) {
ErrorReporter.report(e);
src/main/java/net/coreprotect/consumer/Queue.java
+71 -5
View File
@@ -45,6 +45,8 @@ import net.coreprotect.utility.WorldUtils;
public class Queue {
private static final int MAX_ROLLBACK_ROWS_PER_RECORD = 100_000;
public static synchronized void addForceContainer(String id, ItemStack[] container) {
if (container == null) {
return;
@@ -108,9 +110,16 @@ public class Queue {
}
private static synchronized <T> void queueStandardData(Object[] data, String[] user, Object object, boolean first, Map<Integer, ? extends Map<Integer, T>> additionalMaps, T additionalData, long reservation) {
boolean rollbackPublication = Process.isRollbackPublication((int) data[1], object);
if (Consumer.isPersistenceHalted()) {
Consumer.completeReservation(reservation, 1);
if (rollbackPublication) {
throw new IllegalStateException("Database persistence halted before rollback state could be queued");
}
return;
}
int currentConsumer = (int) (reservation >>> 32);
int consumerId = (int) reservation;
boolean rollbackPublication = Process.isRollbackPublication((int) data[1], object);
boolean published = false;
if (rollbackPublication) {
Consumer.registerRollbackPublications(1);
@@ -344,7 +353,6 @@ public class Queue {
if (user == null || user.trim().isEmpty() || entity == null || action == null) {
return;
}
Location currentLocation = entity.getLocation();
EntityInteractionOrigin origin = EntitySpawnTracking.getOrCreateInteractionOrigin(entity);
if (origin == null || currentLocation.getWorld() == null) {
@@ -377,7 +385,11 @@ public class Queue {
}
public static void queueEntitySpawnLog(String user, java.util.UUID uuid, EntityType type, Location location) {
queueStandardData(new Object[] { null, Process.ENTITY_SPAWN_LOG, null, 0, null, 0, 0, null }, new String[] { user, null }, EntitySpawnData.log(uuid, type, location), false, Consumer.reserveConsumer());
queueEntitySpawnLog(user, EntitySpawnData.log(uuid, type, location));
}
public static void queueEntitySpawnLog(String user, EntitySpawnData spawnData) {
queueStandardData(new Object[] { null, Process.ENTITY_SPAWN_LOG, null, 0, null, 0, 0, null }, new String[] { user, null }, spawnData, false, Consumer.reserveConsumer());
}
public static void queueEntitySpawnLocation(java.util.UUID uuid, Location location, long verificationEpoch) {
@@ -447,14 +459,14 @@ public class Queue {
location = new Location(Bukkit.getServer().getWorlds().get(0), 0, 0, 0);
}
queueStandardData(new Object[] { null, table, null, 0, null, 0, action, null }, new String[] { user, null }, location, false, Consumer.consumerObjectArrayList, list, Consumer.reserveConsumer());
queueRollbackUpdates(user, location, list, table, action, false);
}
public static void queueEntityContainerRollbackUpdate(String user, Location location, List<Object[]> rows, int rollbackType, boolean inventoryRollback) {
if (location == null) {
location = new Location(Bukkit.getServer().getWorlds().get(0), 0, 0, 0);
}
queueStandardData(new Object[] { null, Process.ENTITY_CONTAINER_ROLLBACK_UPDATE, null, inventoryRollback ? 1 : 0, null, 0, rollbackType, null }, new String[] { user, null }, location, false, Consumer.consumerObjectArrayList, rows, Consumer.reserveConsumer());
queueRollbackUpdates(user, location, rows, Process.ENTITY_CONTAINER_ROLLBACK_UPDATE, rollbackType, inventoryRollback);
}
public static void queueEntityContainerRollbackUpdate(String user, EntitySpawnData transition, List<Object[]> rows, int rollbackType, boolean inventoryRollback) {
@@ -463,6 +475,9 @@ public class Queue {
}
public static synchronized void queueEntityRetriesFirst(List<EntityContainerRollbackUpdate> containerUpdates, List<EntitySpawnData> spawnUpdates) {
if (Consumer.isPersistenceHalted()) {
return;
}
List<EntityContainerRollbackUpdate> containerRetries = new ArrayList<>(containerUpdates);
List<EntitySpawnData> spawnRetries = new ArrayList<>(spawnUpdates);
int updateCount = containerRetries.size() + spawnRetries.size();
@@ -519,6 +534,57 @@ public class Queue {
}
}
private static synchronized void queueRollbackUpdates(String user, Location location, List<Object[]> rows, int table, int action, boolean inventoryRollback) {
Objects.requireNonNull(rows, "rows");
if (rows.isEmpty()) {
return;
}
int recordCount = ((rows.size() - 1) / MAX_ROLLBACK_ROWS_PER_RECORD) + 1;
long reservation = Consumer.reserveConsumers(recordCount);
int currentConsumer = (int) (reservation >>> 32);
int firstConsumerId = (int) reservation;
List<Object[]> records = new ArrayList<>(recordCount);
boolean registered = false;
boolean published = false;
try {
if (Consumer.isPersistenceHalted()) {
throw new IllegalStateException("Database persistence halted before rollback state could be queued");
}
Consumer.registerRollbackPublications(recordCount);
registered = true;
for (int recordIndex = 0; recordIndex < recordCount; recordIndex++) {
int fromIndex = recordIndex * MAX_ROLLBACK_ROWS_PER_RECORD;
int toIndex = Math.min(fromIndex + MAX_ROLLBACK_ROWS_PER_RECORD, rows.size());
int consumerId = firstConsumerId + recordIndex;
records.add(new Object[] { consumerId, table, null, inventoryRollback ? 1 : 0, null, 0, action, null });
Consumer.consumerUsers.get(currentConsumer).put(consumerId, new String[] { user, null });
Consumer.consumerObjects.get(currentConsumer).put(consumerId, location);
Consumer.consumerObjectArrayList.get(currentConsumer).put(consumerId, new ArrayList<>(rows.subList(fromIndex, toIndex)));
}
Consumer.consumer.get(currentConsumer).addAll(records);
published = true;
}
finally {
try {
if (!published) {
Consumer.consumer.get(currentConsumer).removeAll(records);
for (int recordIndex = 0; recordIndex < recordCount; recordIndex++) {
int consumerId = firstConsumerId + recordIndex;
Consumer.consumerUsers.get(currentConsumer).remove(consumerId);
Consumer.consumerObjects.get(currentConsumer).remove(consumerId);
Consumer.consumerObjectArrayList.get(currentConsumer).remove(consumerId);
}
if (registered) {
Consumer.completeRollbackPublications(recordCount);
}
}
}
finally {
Consumer.completeReservation(reservation, recordCount);
}
}
}
protected static void queueSignText(String user, Location location, int action, int color, int colorSecondary, boolean frontGlowing, boolean backGlowing, boolean isWaxed, boolean isFront, String line1, String line2, String line3, String line4, String line5, String line6, String line7, String line8, int offset) {
/*
if (line1.length() == 0 && line2.length() == 0 && line3.length() == 0 && line4.length() == 0) {
src/main/java/net/coreprotect/consumer/process/ArtInsertProcess.java
+5 -5
View File
@@ -1,9 +1,9 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import java.sql.Statement;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.MaterialStatement;
import net.coreprotect.language.Phrase;
import net.coreprotect.language.Selector;
@@ -11,12 +11,12 @@ import net.coreprotect.utility.Chat;
class ArtInsertProcess {
static void process(PreparedStatement preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
static void process(ConsumerWriteBatch preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
if (name instanceof String) {
String query = "SELECT id FROM " + ConfigHandler.prefix + "art_map WHERE id = '" + materialId + "' LIMIT 0, 1";
boolean hasMaterial = MaterialStatement.hasMaterial(statement, query);
String query = "SELECT id FROM " + ConfigHandler.prefix + "art_map WHERE id = " + materialId + " LIMIT 1 OFFSET 0";
boolean hasMaterial = !ConfigHandler.databaseType.isClickHouse() && MaterialStatement.hasMaterial(statement, query);
if (!hasMaterial) {
MaterialStatement.insert(preparedStmt, batchCount, materialId, (String) name);
MaterialStatement.insert(preparedStmt, ConsumerWriteBatch.ReferenceKind.ART, batchCount, materialId, (String) name);
// validate ID maps to ensure mapping wasn't reloaded from database prior to this insertion completing
ConfigHandler.art.put((String) name, materialId);
src/main/java/net/coreprotect/consumer/process/BlockBreakProcess.java
+2 -2
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.List;
import org.bukkit.Material;
@@ -14,7 +14,7 @@ import net.coreprotect.utility.MaterialUtils;
class BlockBreakProcess {
static void process(PreparedStatement preparedStmt, PreparedStatement preparedStmtSkulls, int batchCount, int processId, int id, Material blockType, int blockDataId, Material replaceType, int forceData, String user, Object object, String blockData) {
static void process(ConsumerWriteBatch preparedStmt, ConsumerWriteBatch preparedStmtSkulls, int batchCount, int processId, int id, Material blockType, int blockDataId, Material replaceType, int forceData, String user, Object object, String blockData) {
if (object instanceof BlockState) {
BlockState block = (BlockState) object;
List<Object> meta = BlockUtils.processMeta(block);
src/main/java/net/coreprotect/consumer/process/BlockDataInsertProcess.java
+5 -5
View File
@@ -1,9 +1,9 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import java.sql.Statement;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.MaterialStatement;
import net.coreprotect.language.Phrase;
import net.coreprotect.language.Selector;
@@ -11,12 +11,12 @@ import net.coreprotect.utility.Chat;
class BlockDataInsertProcess {
static void process(PreparedStatement preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
static void process(ConsumerWriteBatch preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
if (name instanceof String) {
String query = "SELECT id FROM " + ConfigHandler.prefix + "blockdata_map WHERE id = '" + materialId + "' LIMIT 0, 1";
boolean hasMaterial = MaterialStatement.hasMaterial(statement, query);
String query = "SELECT id FROM " + ConfigHandler.prefix + "blockdata_map WHERE id = " + materialId + " LIMIT 1 OFFSET 0";
boolean hasMaterial = !ConfigHandler.databaseType.isClickHouse() && MaterialStatement.hasMaterial(statement, query);
if (!hasMaterial) {
MaterialStatement.insert(preparedStmt, batchCount, materialId, (String) name);
MaterialStatement.insert(preparedStmt, ConsumerWriteBatch.ReferenceKind.BLOCK_DATA, batchCount, materialId, (String) name);
// validate ID maps to ensure mapping wasn't reloaded from database prior to this insertion completing
ConfigHandler.blockdata.put((String) name, materialId);
src/main/java/net/coreprotect/consumer/process/BlockPlaceProcess.java
+2 -2
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.List;
import org.bukkit.Material;
@@ -13,7 +13,7 @@ import net.coreprotect.utility.MaterialUtils;
class BlockPlaceProcess {
static void process(PreparedStatement preparedStmt, PreparedStatement preparedStmtSkulls, int batchCount, Material blockType, int blockData, Material replaceType, int replaceData, int forceData, String user, Object object, String newBlockData, String replacedBlockData) {
static void process(ConsumerWriteBatch preparedStmt, ConsumerWriteBatch preparedStmtSkulls, int batchCount, Material blockType, int blockData, Material replaceType, int replaceData, int forceData, String user, Object object, String newBlockData, String replacedBlockData) {
if (object instanceof BlockState) {
BlockState block = (BlockState) object;
List<Object> meta = BlockUtils.processMeta(block);
src/main/java/net/coreprotect/consumer/process/ContainerBreakProcess.java
+2 -3
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.Map;
import org.bukkit.Location;
@@ -12,14 +12,13 @@ import net.coreprotect.database.logger.ContainerBreakLogger;
class ContainerBreakProcess {
static void process(PreparedStatement preparedStmt, int batchCount, int processId, int id, Material type, String user, Object object) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, int processId, int id, Material type, String user, Object object) {
if (object instanceof Location) {
Location location = (Location) object;
Map<Integer, ItemStack[]> containers = Consumer.consumerContainers.get(processId);
if (containers.get(id) != null) {
ItemStack[] container = containers.get(id);
ContainerBreakLogger.log(preparedStmt, batchCount, user, location, type, container);
containers.remove(id);
}
}
}
src/main/java/net/coreprotect/consumer/process/ContainerTransactionProcess.java
+16 -5
View File
@@ -1,6 +1,5 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
@@ -14,27 +13,39 @@ import org.bukkit.inventory.ItemStack;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.consumer.Queue;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.logger.ContainerLogger;
import net.coreprotect.model.entity.EntityContainerTransaction;
import net.coreprotect.model.entity.EntitySpawnIdentity;
import net.coreprotect.utility.ErrorReporter;
import net.coreprotect.utility.HopperTransactionUtils;
class ContainerTransactionProcess {
static boolean processEntity(PreparedStatement preparedStmtContainer, int batchCount, String user, Object object, EntitySpawnIdentity identity) {
static boolean processEntity(ConsumerWriteBatch preparedStmtContainer, int batchCount, String user, Object object, EntitySpawnIdentity identity) throws Exception {
if (!(object instanceof EntityContainerTransaction) || identity == null) {
return false;
}
ContainerLogger.logEntity(preparedStmtContainer, batchCount, user, identity, (EntityContainerTransaction) object);
if (ConfigHandler.databaseType.isColumnar()) {
preparedStmtContainer.executeAtomically("entity_container_transaction", () -> ContainerLogger.logEntity(preparedStmtContainer, batchCount, user, identity, (EntityContainerTransaction) object));
}
else {
try {
ContainerLogger.logEntity(preparedStmtContainer, batchCount, user, identity, (EntityContainerTransaction) object);
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
return true;
}
static void process(PreparedStatement preparedStmtContainer, PreparedStatement preparedStmtItems, int batchCount, int processId, int id, Material type, int forceData, String user, Object object) {
static void process(ConsumerWriteBatch preparedStmtContainer, ConsumerWriteBatch preparedStmtItems, int batchCount, int processId, int id, Material type, int forceData, String user, Object object) {
if (object instanceof Location) {
Location location = (Location) object;
Map<Integer, Object> inventories = Consumer.consumerInventories.get(processId);
Object inventory = inventories.remove(id);
Object inventory = inventories.get(id);
if (inventory != null) {
String transactingChestId = HopperTransactionUtils.getTransactionId(location);
String loggingChestIdSuffix = HopperTransactionUtils.getLoggingIdSuffix(location);
src/main/java/net/coreprotect/consumer/process/EntityInsertProcess.java
+5 -5
View File
@@ -1,9 +1,9 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import java.sql.Statement;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.MaterialStatement;
import net.coreprotect.language.Phrase;
import net.coreprotect.language.Selector;
@@ -11,12 +11,12 @@ import net.coreprotect.utility.Chat;
class EntityInsertProcess {
static void process(PreparedStatement preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
static void process(ConsumerWriteBatch preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
if (name instanceof String) {
String query = "SELECT id FROM " + ConfigHandler.prefix + "entity_map WHERE id = '" + materialId + "' LIMIT 0, 1";
boolean hasMaterial = MaterialStatement.hasMaterial(statement, query);
String query = "SELECT id FROM " + ConfigHandler.prefix + "entity_map WHERE id = " + materialId + " LIMIT 1 OFFSET 0";
boolean hasMaterial = !ConfigHandler.databaseType.isClickHouse() && MaterialStatement.hasMaterial(statement, query);
if (!hasMaterial) {
MaterialStatement.insert(preparedStmt, batchCount, materialId, (String) name);
MaterialStatement.insert(preparedStmt, ConsumerWriteBatch.ReferenceKind.ENTITY, batchCount, materialId, (String) name);
// validate ID maps to ensure mapping wasn't reloaded from database prior to this insertion completing
ConfigHandler.entities.put((String) name, materialId);
src/main/java/net/coreprotect/consumer/process/EntityKillProcess.java
+2 -3
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.List;
import java.util.Map;
@@ -14,7 +14,7 @@ import net.coreprotect.utility.EntityUtils;
class EntityKillProcess {
static void process(PreparedStatement preparedStmt, PreparedStatement preparedStmtEntities, PreparedStatement preparedStmtEntityKillLinks, int batchCount, int processId, int id, Object object, String user) {
static void process(ConsumerWriteBatch preparedStmt, ConsumerWriteBatch preparedStmtEntities, ConsumerWriteBatch preparedStmtEntityKillLinks, int batchCount, int processId, int id, Object object, String user) {
if (object instanceof Object[]) {
Object[] values = (Object[]) object;
if (values.length <= 1 || !isLocationData(values[0]) || !(values[1] instanceof EntityType)) {
@@ -28,7 +28,6 @@ class EntityKillProcess {
List<Object> objectList = objectLists.get(id);
int entityId = EntityUtils.getEntityId(type);
EntityKillLogger.log(preparedStmt, preparedStmtEntities, preparedStmtEntityKillLinks, batchCount, user, location, objectList, entityId);
objectLists.remove(id);
}
}
}
src/main/java/net/coreprotect/consumer/process/EntitySpawnLogProcess.java
+3 -3
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.logger.EntitySpawnLogger;
import net.coreprotect.model.entity.EntitySpawnData;
@@ -11,10 +11,10 @@ final class EntitySpawnLogProcess {
private EntitySpawnLogProcess() {
}
static EntitySpawnIdentity process(PreparedStatement blockStatement, PreparedStatement entitySpawnStatement, PreparedStatement blockLinkStatement, Object object, String user) {
static EntitySpawnIdentity process(ConsumerWriteBatch batch, Object object, String user) throws Exception {
if (object instanceof EntitySpawnData) {
EntitySpawnData data = (EntitySpawnData) object;
return EntitySpawnLogger.logIdentity(blockStatement, entitySpawnStatement, blockLinkStatement, user, data);
return EntitySpawnLogger.logIdentity(batch, user, data);
}
return null;
}
src/main/java/net/coreprotect/consumer/process/EntitySpawnProcess.java
+1 -1
View File
@@ -16,7 +16,7 @@ class EntitySpawnProcess {
if (object instanceof Object[]) {
BlockState block = (BlockState) ((Object[]) object)[0];
EntityType type = (EntityType) ((Object[]) object)[1];
String query = "SELECT data FROM " + ConfigHandler.prefix + "entity WHERE rowid='" + rowId + "' LIMIT 0, 1";
String query = "SELECT data FROM " + ConfigHandler.prefix + "entity WHERE rowid=" + rowId + " LIMIT 1 OFFSET 0";
List<Object> data = EntityStatement.getData(statement, block, query);
EntityUtil.spawnEntity(block, type, data);
}
src/main/java/net/coreprotect/consumer/process/EntitySpawnUpdateProcess.java
-16
View File
@@ -1,16 +0,0 @@
package net.coreprotect.consumer.process;
import net.coreprotect.database.statement.EntitySpawnStatement;
import net.coreprotect.model.entity.EntitySpawnData;
final class EntitySpawnUpdateProcess {
private EntitySpawnUpdateProcess() {
}
static void process(EntitySpawnStatement.Updates updates, Object object) {
if (object instanceof EntitySpawnData) {
updates.apply((EntitySpawnData) object);
}
}
}
src/main/java/net/coreprotect/consumer/process/ItemTransactionProcess.java
+2 -2
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.Locale;
import org.bukkit.Location;
@@ -11,7 +11,7 @@ import net.coreprotect.database.logger.ItemLogger;
class ItemTransactionProcess extends Queue {
static void process(PreparedStatement preparedStmt, int batchCount, int processId, int id, int forceData, int time, int offset, String user, Object object) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, int processId, int id, int forceData, int time, int offset, String user, Object object) {
if (object instanceof Location) {
Location location = (Location) object;
String loggingItemId = user.toLowerCase(Locale.ROOT) + "." + location.getBlockX() + "." + location.getBlockY() + "." + location.getBlockZ();
src/main/java/net/coreprotect/consumer/process/MaterialInsertProcess.java
+5 -5
View File
@@ -1,9 +1,9 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import java.sql.Statement;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.MaterialStatement;
import net.coreprotect.language.Phrase;
import net.coreprotect.language.Selector;
@@ -11,12 +11,12 @@ import net.coreprotect.utility.Chat;
class MaterialInsertProcess {
static void process(PreparedStatement preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
static void process(ConsumerWriteBatch preparedStmt, Statement statement, int batchCount, Object name, int materialId) {
if (name instanceof String) {
String query = "SELECT id FROM " + ConfigHandler.prefix + "material_map WHERE id = '" + materialId + "' LIMIT 0, 1";
boolean hasMaterial = MaterialStatement.hasMaterial(statement, query);
String query = "SELECT id FROM " + ConfigHandler.prefix + "material_map WHERE id = " + materialId + " LIMIT 1 OFFSET 0";
boolean hasMaterial = !ConfigHandler.databaseType.isClickHouse() && MaterialStatement.hasMaterial(statement, query);
if (!hasMaterial) {
MaterialStatement.insert(preparedStmt, batchCount, materialId, (String) name);
MaterialStatement.insert(preparedStmt, ConsumerWriteBatch.ReferenceKind.MATERIAL, batchCount, materialId, (String) name);
// validate ID maps to ensure mapping wasn't reloaded from database prior to this insertion completing
ConfigHandler.materials.put((String) name, materialId);
src/main/java/net/coreprotect/consumer/process/NaturalBlockBreakProcess.java
+2 -4
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.sql.Statement;
import java.util.List;
import java.util.Map;
@@ -12,11 +12,10 @@ import net.coreprotect.consumer.Consumer;
import net.coreprotect.database.Lookup;
import net.coreprotect.database.logger.BlockBreakLogger;
import net.coreprotect.utility.MaterialUtils;
import net.coreprotect.utility.Util;
class NaturalBlockBreakProcess {
static void process(Statement statement, PreparedStatement preparedStmt, int batchCount, int processId, int id, String user, Object object, Material blockType, int blockData, String overrideData) {
static void process(Statement statement, ConsumerWriteBatch preparedStmt, int batchCount, int processId, int id, String user, Object object, Material blockType, int blockData, String overrideData) {
if (object instanceof BlockState) {
BlockState block = (BlockState) object;
Map<Integer, List<BlockState>> blockLists = Consumer.consumerBlockList.get(processId);
@@ -28,7 +27,6 @@ class NaturalBlockBreakProcess {
user = removed;
}
}
blockLists.remove(id);
BlockBreakLogger.log(preparedStmt, batchCount, user, block.getLocation(), MaterialUtils.getBlockId(blockType), blockData, null, block.getBlockData().getAsString(), overrideData);
}
}
src/main/java/net/coreprotect/consumer/process/PlayerChatProcess.java
+2 -3
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.Map;
import org.bukkit.Location;
@@ -10,7 +10,7 @@ import net.coreprotect.database.logger.ChatLogger;
class PlayerChatProcess {
static void process(PreparedStatement preparedStmt, int batchCount, int processId, int id, Object object, String user) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, int processId, int id, Object object, String user) {
if (!(object instanceof Object[])) {
return;
}
@@ -23,7 +23,6 @@ class PlayerChatProcess {
Long timestamp = (Long) data[0];
Location location = (Location) data[1];
ChatLogger.log(preparedStmt, batchCount, timestamp, location, user, message);
strings.remove(id);
}
}
}
src/main/java/net/coreprotect/consumer/process/PlayerCommandProcess.java
+2 -3
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.Map;
import org.bukkit.Location;
@@ -10,7 +10,7 @@ import net.coreprotect.database.logger.CommandLogger;
class PlayerCommandProcess {
static void process(PreparedStatement preparedStmt, int batchCount, int processId, int id, Object object, String user) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, int processId, int id, Object object, String user) {
if (!(object instanceof Object[])) {
return;
}
@@ -23,7 +23,6 @@ class PlayerCommandProcess {
Long timestamp = (Long) data[0];
Location location = (Location) data[1];
CommandLogger.log(preparedStmt, batchCount, timestamp, location, user, message);
strings.remove(id);
}
}
}
src/main/java/net/coreprotect/consumer/process/PlayerInteractionProcess.java
+2 -2
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import org.bukkit.Material;
import org.bukkit.block.BlockState;
@@ -9,7 +9,7 @@ import net.coreprotect.database.logger.PlayerInteractLogger;
class PlayerInteractionProcess {
static void process(PreparedStatement preparedStmt, int batchCount, String user, Object object, Material type) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, String user, Object object, Material type) {
if (object instanceof BlockState) {
BlockState block = (BlockState) object;
PlayerInteractLogger.log(preparedStmt, batchCount, user, block, type);
src/main/java/net/coreprotect/consumer/process/PlayerKillProcess.java
+2 -2
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import org.bukkit.Location;
import org.bukkit.block.BlockState;
@@ -9,7 +9,7 @@ import net.coreprotect.database.logger.PlayerKillLogger;
class PlayerKillProcess {
static void process(PreparedStatement preparedStmt, int batchCount, int id, Object object, String user) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, int id, Object object, String user) {
if (object instanceof Object[]) {
Object[] values = (Object[]) object;
if (values.length <= 1 || !isLocationData(values[0]) || !(values[1] instanceof String)) {
src/main/java/net/coreprotect/consumer/process/PlayerLoginProcess.java
+4 -6
View File
@@ -1,29 +1,27 @@
package net.coreprotect.consumer.process;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.util.Map;
import org.bukkit.Location;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.logger.PlayerSessionLogger;
import net.coreprotect.database.logger.UsernameLogger;
import net.coreprotect.model.action.SessionActions;
class PlayerLoginProcess {
static void process(Connection connection, PreparedStatement preparedStmt, int batchCount, int processId, int id, Object object, int configSessions, int configUsernames, int time, String user) {
static void process(ConsumerWriteBatch batch, int batchCount, int processId, int id, Object object, int configSessions, int configUsernames, int time, String user) {
if (object instanceof Location) {
Map<Integer, String> strings = Consumer.consumerStrings.get(processId);
if (strings.get(id) != null) {
String uuid = strings.get(id);
Location location = (Location) object;
UsernameLogger.log(connection, user, uuid, configUsernames, time);
UsernameLogger.log(batch, user, uuid, configUsernames, time);
if (configSessions == 1) {
PlayerSessionLogger.log(preparedStmt, batchCount, user, location, time, SessionActions.LOGIN);
PlayerSessionLogger.log(batch, batchCount, user, location, time, SessionActions.LOGIN);
}
strings.remove(id);
}
}
}
src/main/java/net/coreprotect/consumer/process/PlayerLogoutProcess.java
+2 -2
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import org.bukkit.Location;
@@ -9,7 +9,7 @@ import net.coreprotect.model.action.SessionActions;
class PlayerLogoutProcess {
static void process(PreparedStatement preparedStmt, int batchCount, Object object, int time, String user) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, Object object, int time, String user) {
if (object instanceof Location) {
Location location = (Location) object;
PlayerSessionLogger.log(preparedStmt, batchCount, user, location, time, SessionActions.LOGOUT);
src/main/java/net/coreprotect/consumer/process/Process.java
+411 -229
View File
@@ -1,8 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.HashSet;
@@ -17,15 +15,14 @@ import java.util.UUID;
import org.bukkit.Location;
import org.bukkit.Material;
import net.coreprotect.config.Config;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.consumer.Queue;
import net.coreprotect.database.ConsumerEntitySpawnUpdates;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.Database;
import net.coreprotect.database.logger.EntityInteractionLogger;
import net.coreprotect.database.statement.EntityInteractionStatement;
import net.coreprotect.database.statement.EntitySpawnStatement;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.model.entity.EntityContainerRollbackUpdate;
import net.coreprotect.model.entity.EntityContainerTransaction;
import net.coreprotect.model.entity.EntityInteraction;
@@ -75,6 +72,12 @@ public class Process {
public static int lastLockUpdate = 0;
private static volatile int currentConsumerSize = 0;
private enum TransactionOutcome {
COMMITTED,
RETRY,
RETAINED
}
public static int getCurrentConsumerSize() {
return currentConsumerSize;
}
@@ -91,40 +94,50 @@ public class Process {
return operation == EntitySpawnData.Operation.ROLLBACK || operation == EntitySpawnData.Operation.RESTORE || operation == EntitySpawnData.Operation.KILL_ROLLBACK || operation == EntitySpawnData.Operation.KILL_RESTORE || operation == EntitySpawnData.Operation.COMPOSITE_ROLLBACK || operation == EntitySpawnData.Operation.COMPOSITE_RESTORE || operation == EntitySpawnData.Operation.CLAIM_RELEASE;
}
protected static void updateLockTable(Statement statement, int locked) {
protected static void updateLockTable(ConsumerWriteBatch batch, int locked) {
try {
int unixTimestamp = (int) (System.currentTimeMillis() / 1000L);
int timeSinceLastUpdate = unixTimestamp - lastLockUpdate;
if (timeSinceLastUpdate >= 15 || locked == 0) {
statement.executeUpdate("UPDATE " + ConfigHandler.prefix + "database_lock SET status = '" + locked + "', time = '" + unixTimestamp + "' WHERE rowid = '1'");
batch.updateDatabaseLock(locked, unixTimestamp);
lastLockUpdate = unixTimestamp;
}
}
catch (Exception e) {
ErrorReporter.report(e);
Database.handleWriteFailure(e);
}
}
protected static void processConsumer(int processId, boolean lastRun) {
Map<UUID, Location> pendingEntitySpawnLogs = new LinkedHashMap<>();
List<PendingEntitySpawnLog> pendingEntitySpawnLogs = new ArrayList<>();
Map<UUID, EntitySpawnIdentity> entitySpawnIdentities = new LinkedHashMap<>();
Map<UUID, Location> pendingEntityIdentityConfirmations = new LinkedHashMap<>();
Set<UUID> promotedEntityIdentities = new HashSet<>();
List<PendingEntityContainerTransaction> pendingEntityContainerTransactions = new ArrayList<>();
List<PendingEntityInteraction> pendingEntityInteractions = new ArrayList<>();
List<EntityContainerRollbackRetry> pendingEntityContainerRollbacks = new ArrayList<>();
EntitySpawnStatement.Updates entitySpawnUpdates = null;
ConsumerEntitySpawnUpdates entitySpawnUpdates = null;
ArrayList<Object[]> consumerData = null;
Map<Integer, String[]> users = null;
Map<Integer, Object> consumerObject = null;
ConsumerWriteBatch writeBatch = null;
Connection connection = null;
boolean processingStarted = false;
boolean consumerDataCleared = false;
try (Connection connection = Database.getConnection(false, 500)) {
boolean preflightCommitted = false;
int processedThrough = 0;
int attemptedThrough = 0;
try {
connection = Database.getConnection(false, 500);
if (connection == null) {
deferUnavailableColumnarDatabase(processId);
return;
}
Statement statement = connection.createStatement();
Database.performCheckpoint(statement, Config.getGlobal().MYSQL);
ConsumerWriteBatch batch = Database.openConsumerWriteBatch(connection);
writeBatch = batch;
Database.performCheckpoint(statement, ConfigHandler.databaseType);
Consumer.isPaused = true;
consumerData = Consumer.consumer.get(processId);
@@ -133,19 +146,27 @@ public class Process {
int consumerDataSize = consumerData.size();
currentConsumerSize = consumerDataSize;
if (!beginConsumerTransaction(writeBatch)) {
deferConsumerRetry();
return;
}
if (currentConsumerSize == 0) { // No data, skip processing
updateLockTable(statement, (lastRun ? 0 : 1));
updateLockTable(writeBatch, (lastRun ? 0 : 1));
if (!commit(writeBatch)) {
deferConsumerRetry();
return;
}
statement.close();
Consumer.consumer_id.put(processId, new Integer[] { 0, 0 });
Consumer.isPaused = false;
return;
}
boolean hasEntitySpawnLogs = false;
boolean hasEntitySpawnUpdates = false;
boolean hasEntityKills = false;
boolean hasEntityContainerTransactions = false;
boolean hasEntityInteractions = false;
List<EntitySpawnData> entitySpawnUpdateData = new ArrayList<>();
Set<UUID> entityIdentityUuids = new HashSet<>();
Set<Integer> entityIdentityRowIds = new HashSet<>();
for (int index = 0; index < consumerDataSize; index++) {
@@ -154,9 +175,7 @@ public class Process {
continue;
}
int action = (int) data[1];
hasEntitySpawnLogs |= action == Process.ENTITY_SPAWN_LOG;
hasEntitySpawnUpdates |= action == Process.ENTITY_SPAWN_UPDATE || action == Process.ENTITY_CONTAINER_TRANSITION_UPDATE;
hasEntityKills |= action == Process.ENTITY_KILL;
hasEntityContainerTransactions |= action == Process.ENTITY_CONTAINER_TRANSACTION;
hasEntityInteractions |= action == Process.ENTITY_INTERACTION;
@@ -178,6 +197,7 @@ public class Process {
if (spawnData == null) {
continue;
}
entitySpawnUpdateData.add(spawnData);
if (spawnData.getPreviousUuid() != null) {
entityIdentityUuids.add(spawnData.getPreviousUuid());
}
@@ -187,10 +207,6 @@ public class Process {
}
}
if (!beginConsumerTransaction(statement)) {
deferConsumerRetry();
return;
}
// Scan through usernames, ensure everything is loaded in memory.
for (Entry<Integer, String[]> entry : users.entrySet()) {
String[] data = entry.getValue();
@@ -198,17 +214,17 @@ public class Process {
String user = data[0];
String uuid = data[1];
if (user != null && ConfigHandler.playerIdCache.get(user.toLowerCase(Locale.ROOT)) == null) {
UserStatement.loadId(connection, user, uuid);
writeBatch.resolveUserId(user, uuid);
}
}
}
updateLockTable(statement, (lastRun ? 0 : 1));
if (!Database.commitTransactionChecked(statement, Config.getGlobal().MYSQL)) {
Database.rollbackTransaction(statement, Config.getGlobal().MYSQL);
updateLockTable(writeBatch, (lastRun ? 0 : 1));
if (!commit(writeBatch)) {
invalidateUserCaches(users);
deferConsumerRetry();
failConsumerBatch(processId, consumerData, users, consumerObject, 0, 0);
return;
}
preflightCommitted = true;
if (hasEntityContainerTransactions || hasEntityInteractions) {
entitySpawnIdentities.putAll(EntitySpawnStatement.loadIdentities(connection, entityIdentityUuids));
@@ -216,38 +232,18 @@ public class Process {
bindPendingEntitySpawnIdentities(consumerData, consumerObject, entitySpawnIdentities, identitiesByRowId);
}
// Create prepared statements
PreparedStatement preparedStmtSigns = Database.prepareStatement(connection, Database.SIGN, false);
PreparedStatement preparedStmtBlocks = Database.prepareStatement(connection, Database.BLOCK, false);
PreparedStatement preparedStmtSkulls = Database.prepareStatement(connection, Database.SKULL, true);
PreparedStatement preparedStmtContainers = Database.prepareStatement(connection, Database.CONTAINER, false);
PreparedStatement preparedStmtItems = Database.prepareStatement(connection, Database.ITEM, false);
PreparedStatement preparedStmtWorlds = Database.prepareStatement(connection, Database.WORLD, false);
PreparedStatement preparedStmtChat = Database.prepareStatement(connection, Database.CHAT, false);
PreparedStatement preparedStmtCommand = Database.prepareStatement(connection, Database.COMMAND, false);
PreparedStatement preparedStmtSession = Database.prepareStatement(connection, Database.SESSION, false);
PreparedStatement preparedStmtEntities = Database.prepareStatement(connection, Database.ENTITY, true);
PreparedStatement preparedStmtMaterials = Database.prepareStatement(connection, Database.MATERIAL, false);
PreparedStatement preparedStmtArt = Database.prepareStatement(connection, Database.ART, false);
PreparedStatement preparedStmtEntity = Database.prepareStatement(connection, Database.ENTITY_MAP, false);
PreparedStatement preparedStmtBlockdata = Database.prepareStatement(connection, Database.BLOCKDATA, false);
PreparedStatement preparedStmtEntitySpawns = hasEntitySpawnLogs || hasEntityInteractions ? Database.prepareStatement(connection, Database.ENTITY_SPAWN, true) : null;
PreparedStatement preparedStmtEntityContainers = hasEntityContainerTransactions ? Database.prepareStatement(connection, Database.ENTITY_CONTAINER, false) : null;
PreparedStatement preparedStmtEntityInteractions = hasEntityInteractions ? Database.prepareStatement(connection, Database.ENTITY_INTERACTION, false) : null;
PreparedStatement preparedStmtEntityInteractionCheckpoints = hasEntityInteractions ? EntityInteractionStatement.prepareCheckpoint(connection) : null;
PreparedStatement preparedStmtEntitySpawnBlocks = hasEntitySpawnLogs ? Database.prepareStatement(connection, Database.BLOCK, true) : null;
PreparedStatement preparedStmtEntitySpawnLinks = hasEntitySpawnLogs ? EntitySpawnStatement.prepareBlockLink(connection) : null;
PreparedStatement preparedStmtEntityKillLinks = hasEntityKills ? EntitySpawnStatement.prepareKillLink(connection) : null;
entitySpawnUpdates = hasEntitySpawnUpdates ? new EntitySpawnStatement.Updates(connection) : null;
// Scan through consumer data
if (!beginConsumerTransaction(statement)) {
if (!beginConsumerTransaction(writeBatch)) {
deferConsumerRetry();
return;
}
int processedThrough = 0;
entitySpawnUpdates = hasEntitySpawnUpdates ? writeBatch.entitySpawnUpdates() : null;
if (entitySpawnUpdates != null) {
entitySpawnUpdates.prefetch(entitySpawnUpdateData);
}
processingStarted = true;
for (int i = 0; i < consumerDataSize; i++) {
attemptedThrough = i + 1;
Object[] data = consumerData.get(i);
if (data != null) {
int id = (int) data[0];
@@ -257,6 +253,23 @@ public class Process {
Material replaceType = (Material) data[4];
int replaceData = (int) data[5];
int forceData = (int) data[6];
boolean isolatedTransaction = requiresIsolatedDuckDBTransaction(action);
if (isolatedTransaction && i > processedThrough) {
boolean committed = commit(writeBatch);
if (committed) {
processedThrough = i;
}
completeTransactionState(entitySpawnUpdates, pendingEntityContainerTransactions, pendingEntityContainerRollbacks, pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, pendingEntitySpawnLogs, transactionOutcome(committed));
if (!committed) {
failConsumerBatch(processId, consumerData, users, consumerObject, processedThrough, i);
return;
}
if (!beginConsumerTransaction(writeBatch)) {
retryConsumerBatch(processId, consumerData, users, consumerObject, processedThrough);
return;
}
}
if (users.get(id) != null && consumerObject.get(id) != null) {
String user = users.get(id)[0];
@@ -265,28 +278,39 @@ public class Process {
try {
switch (action) {
case Process.BLOCK_BREAK:
BlockBreakProcess.process(preparedStmtBlocks, preparedStmtSkulls, i, processId, id, blockType, blockData, replaceType, forceData, user, object, (String) data[7]);
BlockBreakProcess.process(writeBatch, writeBatch, i, processId, id, blockType, blockData, replaceType, forceData, user, object, (String) data[7]);
break;
case Process.BLOCK_PLACE:
BlockPlaceProcess.process(preparedStmtBlocks, preparedStmtSkulls, i, blockType, blockData, replaceType, replaceData, forceData, user, object, (String) data[7], (String) data[8]);
BlockPlaceProcess.process(writeBatch, writeBatch, i, blockType, blockData, replaceType, replaceData, forceData, user, object, (String) data[7], (String) data[8]);
break;
case Process.SIGN_TEXT:
SignTextProcess.process(preparedStmtSigns, i, processId, id, forceData, user, object, replaceData, blockData);
SignTextProcess.process(writeBatch, i, processId, id, forceData, user, object, replaceData, blockData);
break;
case Process.CONTAINER_BREAK:
ContainerBreakProcess.process(preparedStmtContainers, i, processId, id, blockType, user, object);
ContainerBreakProcess.process(writeBatch, i, processId, id, blockType, user, object);
break;
case Process.PLAYER_INTERACTION:
PlayerInteractionProcess.process(preparedStmtBlocks, i, user, object, blockType);
PlayerInteractionProcess.process(writeBatch, i, user, object, blockType);
break;
case Process.CONTAINER_TRANSACTION:
ContainerTransactionProcess.process(preparedStmtContainers, preparedStmtItems, i, processId, id, blockType, forceData, user, object);
ContainerTransactionProcess.process(writeBatch, writeBatch, i, processId, id, blockType, forceData, user, object);
break;
case Process.ENTITY_CONTAINER_TRANSACTION:
EntityContainerTransaction transaction = (EntityContainerTransaction) object;
EntitySpawnIdentity identity = entitySpawnIdentities.get(transaction.getEntityUuid());
if (!ContainerTransactionProcess.processEntity(preparedStmtEntityContainers, i, user, transaction, identity)) {
retryEntityContainerTransaction(user, transaction);
boolean processedEntityContainerTransaction;
try {
processedEntityContainerTransaction = ContainerTransactionProcess.processEntity(writeBatch, i, user, transaction, identity);
}
catch (Exception e) {
pendingEntityContainerTransactions.add(new PendingEntityContainerTransaction(user, transaction, true));
throw e;
}
if (!processedEntityContainerTransaction) {
pendingEntityContainerTransactions.add(new PendingEntityContainerTransaction(user, transaction, true));
}
else if (ConfigHandler.databaseType.isColumnar()) {
pendingEntityContainerTransactions.add(new PendingEntityContainerTransaction(user, transaction, false));
}
break;
case Process.ENTITY_INTERACTION:
@@ -299,72 +323,66 @@ public class Process {
EntitySpawnIdentity existingIdentity = entitySpawnIdentities.get(interaction.getEntityUuid());
EntitySpawnIdentity[] loggedIdentity = { existingIdentity };
try {
Database.executeSavepoint(statement, "entity_interaction_log", () -> {
writeBatch.executeAtomically("entity_interaction_log", () -> {
if (loggedIdentity[0] == null) {
int time = (int) (System.currentTimeMillis() / 1000L);
loggedIdentity[0] = EntitySpawnStatement.insertIdentity(preparedStmtEntitySpawns, time, interaction.getEntityUuid(), interaction.getOrigin(), interaction.getCurrentLocation());
loggedIdentity[0] = EntitySpawnStatement.insertIdentity(batch, time, interaction.getEntityUuid(), interaction.getOrigin(), interaction.getCurrentLocation());
}
EntityInteractionLogger.log(preparedStmtEntityInteractions, preparedStmtEntityInteractionCheckpoints, loggedIdentity[0], interaction, logContext);
EntityInteractionLogger.log(batch, loggedIdentity[0], interaction, logContext);
});
}
catch (Exception e) {
retryEntityInteraction(user, interaction);
pendingEntityInteractions.add(new PendingEntityInteraction(user, interaction, true));
throw e;
}
entitySpawnIdentities.put(interaction.getEntityUuid(), loggedIdentity[0]);
pendingEntityInteractions.add(new PendingEntityInteraction(user, interaction));
pendingEntityInteractions.add(new PendingEntityInteraction(user, interaction, false));
if (existingIdentity == null) {
promotedEntityIdentities.add(interaction.getEntityUuid());
}
pendingEntityIdentityConfirmations.put(interaction.getEntityUuid(), interaction.getCurrentLocation());
break;
case Process.ITEM_TRANSACTION:
ItemTransactionProcess.process(preparedStmtItems, i, processId, id, forceData, replaceData, blockData, user, object);
ItemTransactionProcess.process(writeBatch, i, processId, id, forceData, replaceData, blockData, user, object);
break;
case Process.STRUCTURE_GROWTH:
StructureGrowthProcess.process(statement, preparedStmtBlocks, i, processId, id, user, object, forceData);
StructureGrowthProcess.process(statement, writeBatch, i, processId, id, user, object, forceData);
break;
case Process.ROLLBACK_UPDATE:
RollbackUpdateProcess.process(statement, processId, id, forceData, RollbackUpdateTargets.BLOCK);
RollbackUpdateProcess.process(writeBatch, processId, id, forceData, RollbackUpdateTargets.BLOCK);
break;
case Process.CONTAINER_ROLLBACK_UPDATE:
RollbackUpdateProcess.process(statement, processId, id, forceData, RollbackUpdateTargets.CONTAINER);
RollbackUpdateProcess.process(writeBatch, processId, id, forceData, RollbackUpdateTargets.CONTAINER);
break;
case Process.INVENTORY_ROLLBACK_UPDATE:
RollbackUpdateProcess.process(statement, processId, id, forceData, RollbackUpdateTargets.INVENTORY_ITEM);
RollbackUpdateProcess.process(writeBatch, processId, id, forceData, RollbackUpdateTargets.INVENTORY_ITEM);
break;
case Process.INVENTORY_CONTAINER_ROLLBACK_UPDATE:
RollbackUpdateProcess.process(statement, processId, id, forceData, RollbackUpdateTargets.INVENTORY_CONTAINER);
RollbackUpdateProcess.process(writeBatch, processId, id, forceData, RollbackUpdateTargets.INVENTORY_CONTAINER);
break;
case Process.BLOCK_INVENTORY_ROLLBACK_UPDATE:
RollbackUpdateProcess.process(statement, processId, id, forceData, RollbackUpdateTargets.BLOCK_INVENTORY);
RollbackUpdateProcess.process(writeBatch, processId, id, forceData, RollbackUpdateTargets.BLOCK_INVENTORY);
break;
case Process.ENTITY_CONTAINER_ROLLBACK_UPDATE:
List<Object[]> entityContainerRows = Consumer.consumerObjectArrayList.get(processId).get(id);
if (entityContainerRows != null) {
try {
RollbackUpdateProcess.processChecked(statement, entityContainerRows, forceData, RollbackUpdateTargets.ENTITY_CONTAINER, blockData == 1);
Consumer.consumerObjectArrayList.get(processId).remove(id);
pendingEntityContainerRollbacks.add(new EntityContainerRollbackRetry(user, object instanceof Location ? (Location) object : null, entityContainerRows, forceData, blockData == 1));
writeBatch.executeAtomically("entity_container_rollback", () -> RollbackUpdateProcess.processChecked(batch, entityContainerRows, forceData, RollbackUpdateTargets.ENTITY_CONTAINER, blockData == 1));
pendingEntityContainerRollbacks.add(new EntityContainerRollbackRetry(user, object instanceof Location ? (Location) object : null, entityContainerRows, forceData, blockData == 1, false));
}
catch (SQLException e) {
try {
Queue.queueEntityContainerRollbackUpdate(user, object instanceof Location ? (Location) object : null, entityContainerRows, forceData, blockData == 1);
}
catch (Exception retryException) {
e.addSuppressed(retryException);
}
catch (Exception e) {
pendingEntityContainerRollbacks.add(new EntityContainerRollbackRetry(user, object instanceof Location ? (Location) object : null, entityContainerRows, forceData, blockData == 1, true));
throw e;
}
}
break;
case Process.ENTITY_CONTAINER_TRANSITION_UPDATE:
EntityContainerRollbackUpdate containerUpdate = (EntityContainerRollbackUpdate) object;
entitySpawnUpdates.applyCombined(containerUpdate, () -> RollbackUpdateProcess.processChecked(statement, containerUpdate.getRows(), containerUpdate.getRollbackType(), RollbackUpdateTargets.ENTITY_CONTAINER, containerUpdate.isInventoryRollback()));
entitySpawnUpdates.applyCombined(containerUpdate, () -> RollbackUpdateProcess.processChecked(batch, containerUpdate.getRows(), containerUpdate.getRollbackType(), RollbackUpdateTargets.ENTITY_CONTAINER, containerUpdate.isInventoryRollback()));
break;
case Process.WORLD_INSERT:
WorldInsertProcess.process(preparedStmtWorlds, i, statement, object, forceData);
WorldInsertProcess.process(writeBatch, i, statement, object, forceData);
break;
case Process.SIGN_UPDATE:
SignUpdateProcess.process(statement, object, user, blockData, forceData);
@@ -373,159 +391,168 @@ public class Process {
SkullUpdateProcess.process(statement, object, forceData);
break;
case Process.PLAYER_CHAT:
PlayerChatProcess.process(preparedStmtChat, i, processId, id, object, user);
PlayerChatProcess.process(writeBatch, i, processId, id, object, user);
break;
case Process.PLAYER_COMMAND:
PlayerCommandProcess.process(preparedStmtCommand, i, processId, id, object, user);
PlayerCommandProcess.process(writeBatch, i, processId, id, object, user);
break;
case Process.PLAYER_LOGIN:
PlayerLoginProcess.process(connection, preparedStmtSession, i, processId, id, object, blockData, replaceData, forceData, user);
PlayerLoginProcess.process(writeBatch, i, processId, id, object, blockData, replaceData, forceData, user);
break;
case Process.PLAYER_LOGOUT:
PlayerLogoutProcess.process(preparedStmtSession, i, object, forceData, user);
PlayerLogoutProcess.process(writeBatch, i, object, forceData, user);
break;
case Process.ENTITY_KILL:
EntityKillProcess.process(preparedStmtBlocks, preparedStmtEntities, preparedStmtEntityKillLinks, i, processId, id, object, user);
EntityKillProcess.process(writeBatch, writeBatch, writeBatch, i, processId, id, object, user);
break;
case Process.ENTITY_SPAWN:
EntitySpawnProcess.process(statement, object, forceData);
break;
case Process.NATURAL_BLOCK_BREAK:
NaturalBlockBreakProcess.process(statement, preparedStmtBlocks, i, processId, id, user, object, blockType, blockData, (String) data[7]);
NaturalBlockBreakProcess.process(statement, writeBatch, i, processId, id, user, object, blockType, blockData, (String) data[7]);
break;
case Process.MATERIAL_INSERT:
MaterialInsertProcess.process(preparedStmtMaterials, statement, i, object, forceData);
MaterialInsertProcess.process(writeBatch, statement, i, object, forceData);
break;
case Process.ART_INSERT:
ArtInsertProcess.process(preparedStmtArt, statement, i, object, forceData);
ArtInsertProcess.process(writeBatch, statement, i, object, forceData);
break;
case Process.ENTITY_INSERT:
EntityInsertProcess.process(preparedStmtEntity, statement, i, object, forceData);
EntityInsertProcess.process(writeBatch, statement, i, object, forceData);
break;
case Process.PLAYER_KILL:
PlayerKillProcess.process(preparedStmtBlocks, i, id, object, user);
PlayerKillProcess.process(writeBatch, i, id, object, user);
break;
case Process.BLOCKDATA_INSERT:
BlockDataInsertProcess.process(preparedStmtBlockdata, statement, i, object, forceData);
BlockDataInsertProcess.process(writeBatch, statement, i, object, forceData);
break;
case Process.ENTITY_SPAWN_LOG:
EntitySpawnIdentity spawnIdentity = EntitySpawnLogProcess.process(preparedStmtEntitySpawnBlocks, preparedStmtEntitySpawns, preparedStmtEntitySpawnLinks, object, user);
if (spawnIdentity != null && object instanceof EntitySpawnData) {
if (!(object instanceof EntitySpawnData)) {
break;
}
EntitySpawnData spawnData = (EntitySpawnData) object;
EntitySpawnIdentity spawnIdentity;
try {
spawnIdentity = EntitySpawnLogProcess.process(writeBatch, spawnData, user);
}
catch (Exception e) {
if (ConfigHandler.databaseType.isColumnar()) {
pendingEntitySpawnLogs.add(new PendingEntitySpawnLog(user, spawnData, false, true));
}
else {
EntitySpawnTracking.clearTracking(spawnData.getUuid());
}
throw e;
}
if (spawnIdentity != null) {
entitySpawnIdentities.put(spawnIdentity.getUuid(), spawnIdentity);
pendingEntitySpawnLogs.put(spawnIdentity.getUuid(), ((EntitySpawnData) object).getLocation());
pendingEntitySpawnLogs.add(new PendingEntitySpawnLog(user, spawnData, true, false));
}
break;
case Process.ENTITY_SPAWN_UPDATE:
EntitySpawnUpdateProcess.process(entitySpawnUpdates, object);
if (object instanceof EntitySpawnData) {
entitySpawnUpdates.apply((EntitySpawnData) object);
}
break;
}
// If database connection goes missing, remove processed data from consumer and abort
if (statement.isClosed()) {
if (entitySpawnUpdates != null) {
entitySpawnUpdates.afterCommit(false);
}
retryEntityContainerRollbacks(pendingEntityContainerRollbacks, false);
completeEntityInteractions(pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, false);
discardProcessedConsumerData(processId, consumerData, users, consumerObject, i + 1);
deferConsumerRetry();
completeEntitySpawnLogs(pendingEntitySpawnLogs, false);
return;
}
// If interrupt requested, commit data, sleep, and resume processing
if (Consumer.interrupt) {
boolean committed = commit(statement, preparedStmtSigns, preparedStmtBlocks, preparedStmtSkulls, preparedStmtContainers, preparedStmtEntityContainers, preparedStmtItems, preparedStmtWorlds, preparedStmtChat, preparedStmtCommand, preparedStmtSession, preparedStmtEntities, preparedStmtMaterials, preparedStmtArt, preparedStmtEntity, preparedStmtBlockdata, preparedStmtEntityKillLinks);
if (entitySpawnUpdates != null) {
entitySpawnUpdates.afterCommit(committed);
boolean interrupted = Consumer.interrupt;
if ((interrupted || writeBatch.shouldCommit()) && !isolatedTransaction) {
boolean committed = commit(writeBatch);
if (committed) {
processedThrough = i + 1;
}
retryEntityContainerRollbacks(pendingEntityContainerRollbacks, committed);
completeEntityInteractions(pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, committed);
completeEntitySpawnLogs(pendingEntitySpawnLogs, committed);
processedThrough = i + 1;
Thread.sleep(500);
if (!beginConsumerTransaction(statement)) {
discardProcessedConsumerData(processId, consumerData, users, consumerObject, processedThrough);
deferConsumerRetry();
completeTransactionState(entitySpawnUpdates, pendingEntityContainerTransactions, pendingEntityContainerRollbacks, pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, pendingEntitySpawnLogs, transactionOutcome(committed));
if (!committed) {
failConsumerBatch(processId, consumerData, users, consumerObject, processedThrough, i + 1);
return;
}
if (interrupted) {
try {
Thread.sleep(500);
}
catch (InterruptedException exception) {
Thread.currentThread().interrupt();
retryConsumerBatch(processId, consumerData, users, consumerObject, processedThrough);
return;
}
}
if (!beginConsumerTransaction(writeBatch)) {
retryConsumerBatch(processId, consumerData, users, consumerObject, processedThrough);
return;
}
}
}
catch (Exception e) {
if (ConfigHandler.databaseType.isColumnar()) {
throw e;
}
ErrorReporter.report(e);
}
// If database connection goes missing, abort and roll back the transaction
if (statement.isClosed()) {
throw new IllegalStateException("Database connection closed during consumer processing");
}
}
if (isolatedTransaction) {
boolean committed = commit(writeBatch);
if (committed) {
processedThrough = i + 1;
}
completeTransactionState(entitySpawnUpdates, pendingEntityContainerTransactions, pendingEntityContainerRollbacks, pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, pendingEntitySpawnLogs, transactionOutcome(committed));
if (!committed) {
failConsumerBatch(processId, consumerData, users, consumerObject, processedThrough, i + 1);
return;
}
if (!beginConsumerTransaction(writeBatch)) {
retryConsumerBatch(processId, consumerData, users, consumerObject, processedThrough);
return;
}
}
}
currentConsumerSize--;
}
// commit data to database
boolean committed = commit(statement, preparedStmtSigns, preparedStmtBlocks, preparedStmtSkulls, preparedStmtContainers, preparedStmtEntityContainers, preparedStmtItems, preparedStmtWorlds, preparedStmtChat, preparedStmtCommand, preparedStmtSession, preparedStmtEntities, preparedStmtMaterials, preparedStmtArt, preparedStmtEntity, preparedStmtBlockdata, preparedStmtEntityKillLinks);
try {
try {
if (entitySpawnUpdates != null) {
entitySpawnUpdates.afterCommit(committed);
}
}
finally {
retryEntityContainerRollbacks(pendingEntityContainerRollbacks, committed);
completeEntityInteractions(pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, committed);
completeEntitySpawnLogs(pendingEntitySpawnLogs, committed);
}
boolean committed = commit(writeBatch);
if (committed) {
processedThrough = consumerData.size();
}
finally {
clearConsumerData(processId, consumerData, users, consumerObject);
consumerDataCleared = true;
completeTransactionState(entitySpawnUpdates, pendingEntityContainerTransactions, pendingEntityContainerRollbacks, pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, pendingEntitySpawnLogs, transactionOutcome(committed));
if (!committed) {
failConsumerBatch(processId, consumerData, users, consumerObject, processedThrough, consumerData.size());
return;
}
clearConsumerData(processId, consumerData, users, consumerObject);
consumerDataCleared = true;
// close connections/statements
preparedStmtSigns.close();
preparedStmtBlocks.close();
preparedStmtSkulls.close();
preparedStmtContainers.close();
if (preparedStmtEntityContainers != null) {
preparedStmtEntityContainers.close();
}
if (preparedStmtEntityInteractions != null) {
preparedStmtEntityInteractions.close();
preparedStmtEntityInteractionCheckpoints.close();
}
preparedStmtItems.close();
preparedStmtWorlds.close();
preparedStmtChat.close();
preparedStmtCommand.close();
preparedStmtSession.close();
preparedStmtEntities.close();
preparedStmtMaterials.close();
preparedStmtArt.close();
preparedStmtEntity.close();
preparedStmtBlockdata.close();
if (preparedStmtEntitySpawns != null) {
preparedStmtEntitySpawns.close();
if (preparedStmtEntitySpawnBlocks != null) {
preparedStmtEntitySpawnBlocks.close();
preparedStmtEntitySpawnLinks.close();
}
}
if (preparedStmtEntityKillLinks != null) {
preparedStmtEntityKillLinks.close();
}
if (entitySpawnUpdates != null) {
entitySpawnUpdates.close();
entitySpawnUpdates = null;
}
statement.close();
}
catch (Exception e) {
if (processingStarted && !consumerDataCleared && consumerData != null && users != null && consumerObject != null) {
if (writeBatch != null && Consumer.transacting) {
writeBatch.rollback();
}
if (!preflightCommitted && users != null) {
invalidateUserCaches(users);
}
if ((processingStarted || ConfigHandler.databaseType.isColumnar()) && !consumerDataCleared && consumerData != null && users != null && consumerObject != null) {
try {
if (entitySpawnUpdates != null) {
entitySpawnUpdates.afterCommit(false);
completeTransactionState(entitySpawnUpdates, pendingEntityContainerTransactions, pendingEntityContainerRollbacks, pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, pendingEntitySpawnLogs, failedTransactionOutcome());
if (ConfigHandler.databaseType.isColumnar()) {
discardProcessedConsumerData(processId, consumerData, users, consumerObject, processedThrough);
currentConsumerSize = consumerData.size();
if (processingStarted) {
Consumer.haltPersistence();
}
}
retryEntityContainerRollbacks(pendingEntityContainerRollbacks, false);
completeEntityInteractions(pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, false);
clearConsumerData(processId, consumerData, users, consumerObject);
consumerDataCleared = true;
else {
discardProcessedConsumerData(processId, consumerData, users, consumerObject, processedThrough);
discardProcessedConsumerData(processId, consumerData, users, consumerObject, Math.max(0, attemptedThrough - processedThrough));
}
consumerDataCleared = consumerData.isEmpty();
}
catch (Exception cleanupException) {
e.addSuppressed(cleanupException);
@@ -534,24 +561,33 @@ public class Process {
ErrorReporter.report(e);
}
finally {
if (entitySpawnUpdates != null) {
if (writeBatch != null) {
try {
entitySpawnUpdates.close();
writeBatch.close();
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
if (connection != null) {
try {
connection.close();
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
}
completeEntityInteractions(pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, false);
completeEntitySpawnLogs(pendingEntitySpawnLogs, false);
TransactionOutcome cleanupOutcome = failedTransactionOutcome();
completeEntityInteractions(pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, cleanupOutcome);
completeEntitySpawnLogs(pendingEntitySpawnLogs, cleanupOutcome);
if (consumerDataCleared) {
currentConsumerSize = 0;
Consumer.consumer_id.put(processId, new Integer[] { 0, 0 });
Consumer.isPaused = false;
}
else {
else if (!Consumer.isPersistenceHalted()) {
deferConsumerRetry();
}
}
@@ -615,14 +651,22 @@ public class Process {
}
}
private static void completeEntityInteractions(List<PendingEntityInteraction> interactions, Map<UUID, Location> identityConfirmations, Set<UUID> promotedIdentities, Map<UUID, EntitySpawnIdentity> identities, boolean committed) {
if (committed) {
private static void completeEntityInteractions(List<PendingEntityInteraction> interactions, Map<UUID, Location> identityConfirmations, Set<UUID> promotedIdentities, Map<UUID, EntitySpawnIdentity> identities, TransactionOutcome outcome) {
if (outcome == TransactionOutcome.COMMITTED) {
for (Map.Entry<UUID, Location> entry : identityConfirmations.entrySet()) {
EntitySpawnTracking.confirmDatabaseIdentity(entry.getKey(), entry.getValue());
try {
EntitySpawnTracking.confirmDatabaseIdentity(entry.getKey(), entry.getValue());
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
}
else {
if (outcome != TransactionOutcome.RETAINED) {
for (PendingEntityInteraction pending : interactions) {
if (outcome == TransactionOutcome.COMMITTED && !pending.retryRequired) {
continue;
}
try {
retryEntityInteraction(pending.user, pending.interaction);
}
@@ -630,6 +674,8 @@ public class Process {
ErrorReporter.report(e);
}
}
}
if (outcome != TransactionOutcome.COMMITTED) {
for (UUID uuid : promotedIdentities) {
identities.remove(uuid);
}
@@ -639,9 +685,12 @@ public class Process {
promotedIdentities.clear();
}
private static void retryEntityContainerRollbacks(List<EntityContainerRollbackRetry> updates, boolean committed) {
if (!committed) {
private static void retryEntityContainerRollbacks(List<EntityContainerRollbackRetry> updates, TransactionOutcome outcome) {
if (outcome != TransactionOutcome.RETAINED) {
for (EntityContainerRollbackRetry update : updates) {
if (outcome == TransactionOutcome.COMMITTED && !update.retryRequired) {
continue;
}
try {
Queue.queueEntityContainerRollbackUpdate(update.user, update.location, update.rows, update.rollbackType, update.inventoryRollback);
}
@@ -653,9 +702,9 @@ public class Process {
updates.clear();
}
private static boolean beginConsumerTransaction(Statement statement) {
private static boolean beginConsumerTransaction(ConsumerWriteBatch batch) {
try {
Database.beginTransaction(statement, Config.getGlobal().MYSQL);
batch.begin();
return true;
}
catch (Exception e) {
@@ -669,6 +718,35 @@ public class Process {
Consumer.isPaused = false;
}
static void retryConsumerBatch(int processId, ArrayList<Object[]> consumerData, Map<Integer, String[]> users, Map<Integer, Object> consumerObject, int processedThrough) {
discardProcessedConsumerData(processId, consumerData, users, consumerObject, processedThrough);
deferConsumerRetry();
}
private static void deferUnavailableColumnarDatabase(int processId) {
if (!ConfigHandler.databaseType.isColumnar()) {
return;
}
ArrayList<Object[]> consumerData = Consumer.consumer.get(processId);
if (consumerData != null) {
currentConsumerSize = consumerData.size();
}
Consumer.isPaused = false;
}
static void failConsumerBatch(int processId, ArrayList<Object[]> consumerData, Map<Integer, String[]> users, Map<Integer, Object> consumerObject, int processedThrough, int legacyDiscardThrough) {
if (ConfigHandler.databaseType.isColumnar()) {
discardProcessedConsumerData(processId, consumerData, users, consumerObject, processedThrough);
currentConsumerSize = consumerData.size();
Consumer.haltPersistence();
}
else {
discardProcessedConsumerData(processId, consumerData, users, consumerObject, processedThrough);
discardProcessedConsumerData(processId, consumerData, users, consumerObject, Math.max(0, legacyDiscardThrough - processedThrough));
deferConsumerRetry();
}
}
private static void invalidateUserCaches(Map<Integer, String[]> users) {
for (String[] data : users.values()) {
if (data == null || data[0] == null) {
@@ -717,14 +795,47 @@ public class Process {
consumerObject.clear();
}
private static void completeEntitySpawnLogs(Map<UUID, Location> pendingEntitySpawnLogs, boolean committed) {
for (Map.Entry<UUID, Location> entry : pendingEntitySpawnLogs.entrySet()) {
if (committed) {
EntitySpawnTracking.confirmDatabaseIdentity(entry.getKey(), entry.getValue());
}
else {
private static void completeEntityContainerTransactions(List<PendingEntityContainerTransaction> transactions, TransactionOutcome outcome) {
if (outcome != TransactionOutcome.RETAINED) {
for (PendingEntityContainerTransaction pending : transactions) {
if (outcome == TransactionOutcome.COMMITTED && !pending.retryRequired) {
continue;
}
try {
EntitySpawnTracking.reverifyDatabaseRow(entry.getKey(), entry.getValue());
retryEntityContainerTransaction(pending.user, pending.transaction);
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
}
transactions.clear();
}
private static void completeEntitySpawnLogs(List<PendingEntitySpawnLog> pendingEntitySpawnLogs, TransactionOutcome outcome) {
for (PendingEntitySpawnLog pending : pendingEntitySpawnLogs) {
EntitySpawnData spawnData = pending.data;
if (outcome == TransactionOutcome.COMMITTED) {
if (pending.retryRequired) {
retryEntitySpawnLog(pending);
}
else {
try {
if (pending.confirmIdentity) {
EntitySpawnTracking.confirmDatabaseIdentity(spawnData.getUuid(), spawnData.getLocation());
}
else {
EntitySpawnTracking.clearTracking(spawnData.getUuid());
}
}
catch (Exception e) {
ErrorReporter.report(e);
}
}
}
else if (outcome == TransactionOutcome.RETRY) {
try {
EntitySpawnTracking.reverifyDatabaseRow(spawnData.getUuid(), spawnData.getLocation());
}
catch (Exception e) {
ErrorReporter.report(e);
@@ -734,41 +845,93 @@ public class Process {
pendingEntitySpawnLogs.clear();
}
private static boolean commit(Statement statement, PreparedStatement preparedStmtSigns, PreparedStatement preparedStmtBlocks, PreparedStatement preparedStmtSkulls, PreparedStatement preparedStmtContainers, PreparedStatement preparedStmtEntityContainers, PreparedStatement preparedStmtItems, PreparedStatement preparedStmtWorlds, PreparedStatement preparedStmtChat, PreparedStatement preparedStmtCommand, PreparedStatement preparedStmtSession, PreparedStatement preparedStmtEntities, PreparedStatement preparedStmtMaterials, PreparedStatement preparedStmtArt, PreparedStatement preparedStmtEntity, PreparedStatement preparedStmtBlockdata, PreparedStatement preparedStmtEntityKillLinks) {
private static void retryEntitySpawnLog(PendingEntitySpawnLog pending) {
EntitySpawnData spawnData = pending.data;
try {
preparedStmtSigns.executeBatch();
preparedStmtBlocks.executeBatch();
preparedStmtSkulls.executeBatch();
preparedStmtContainers.executeBatch();
if (preparedStmtEntityContainers != null) {
preparedStmtEntityContainers.executeBatch();
EntitySpawnData retry = spawnData.retryLog();
if (retry != null) {
Queue.queueEntitySpawnLog(pending.user, retry);
}
preparedStmtItems.executeBatch();
preparedStmtWorlds.executeBatch();
preparedStmtChat.executeBatch();
preparedStmtCommand.executeBatch();
preparedStmtSession.executeBatch();
preparedStmtEntities.executeBatch();
preparedStmtMaterials.executeBatch();
preparedStmtArt.executeBatch();
preparedStmtEntity.executeBatch();
preparedStmtBlockdata.executeBatch();
if (preparedStmtEntityKillLinks != null) {
preparedStmtEntityKillLinks.executeBatch();
else {
EntitySpawnTracking.clearTracking(spawnData.getUuid());
ErrorReporter.report(new IllegalStateException("Dropped entity spawn log after repeated persistence failures: " + spawnData.getUuid()), ConfigHandler.EDITION_BRANCH.contains("-dev"));
}
boolean committed = Database.commitTransactionChecked(statement, Config.getGlobal().MYSQL);
if (!committed) {
Database.rollbackTransaction(statement, Config.getGlobal().MYSQL);
}
return committed;
}
catch (Exception e) {
Database.rollbackTransaction(statement, Config.getGlobal().MYSQL);
ErrorReporter.report(e);
}
}
private static boolean requiresIsolatedDuckDBTransaction(int action) {
if (!ConfigHandler.databaseType.isDuckDB()) {
return false;
}
return action == ENTITY_CONTAINER_TRANSACTION || action == ENTITY_INTERACTION || action == ENTITY_CONTAINER_ROLLBACK_UPDATE || action == ENTITY_CONTAINER_TRANSITION_UPDATE || action == ENTITY_SPAWN_LOG || action == ENTITY_SPAWN_UPDATE;
}
private static TransactionOutcome transactionOutcome(boolean committed) {
return committed ? TransactionOutcome.COMMITTED : failedTransactionOutcome();
}
private static TransactionOutcome failedTransactionOutcome() {
return ConfigHandler.databaseType.isColumnar() ? TransactionOutcome.RETAINED : TransactionOutcome.RETRY;
}
private static void completeTransactionState(ConsumerEntitySpawnUpdates entitySpawnUpdates, List<PendingEntityContainerTransaction> pendingEntityContainerTransactions, List<EntityContainerRollbackRetry> pendingEntityContainerRollbacks, List<PendingEntityInteraction> pendingEntityInteractions, Map<UUID, Location> pendingEntityIdentityConfirmations, Set<UUID> promotedEntityIdentities, Map<UUID, EntitySpawnIdentity> entitySpawnIdentities, List<PendingEntitySpawnLog> pendingEntitySpawnLogs, TransactionOutcome outcome) {
try {
if (entitySpawnUpdates != null) {
if (outcome == TransactionOutcome.RETAINED) {
entitySpawnUpdates.afterRetain();
}
else {
entitySpawnUpdates.afterCommit(outcome == TransactionOutcome.COMMITTED);
}
}
}
finally {
try {
completeEntityContainerTransactions(pendingEntityContainerTransactions, outcome);
}
finally {
try {
retryEntityContainerRollbacks(pendingEntityContainerRollbacks, outcome);
}
finally {
try {
completeEntityInteractions(pendingEntityInteractions, pendingEntityIdentityConfirmations, promotedEntityIdentities, entitySpawnIdentities, outcome);
}
finally {
completeEntitySpawnLogs(pendingEntitySpawnLogs, outcome);
}
}
}
}
}
private static boolean commit(ConsumerWriteBatch batch) {
try {
return batch.commit();
}
catch (Exception e) {
batch.rollback();
ErrorReporter.report(e);
return false;
}
}
private static final class PendingEntityContainerTransaction {
private final String user;
private final EntityContainerTransaction transaction;
private final boolean retryRequired;
private PendingEntityContainerTransaction(String user, EntityContainerTransaction transaction, boolean retryRequired) {
this.user = user;
this.transaction = transaction;
this.retryRequired = retryRequired;
}
}
private static final class EntityContainerRollbackRetry {
private final String user;
@@ -776,8 +939,9 @@ public class Process {
private final List<Object[]> rows;
private final int rollbackType;
private final boolean inventoryRollback;
private final boolean retryRequired;
private EntityContainerRollbackRetry(String user, Location location, List<Object[]> rows, int rollbackType, boolean inventoryRollback) {
private EntityContainerRollbackRetry(String user, Location location, List<Object[]> rows, int rollbackType, boolean inventoryRollback, boolean retryRequired) {
this.user = user;
this.location = location == null ? null : location.clone();
this.rows = new ArrayList<>(rows.size());
@@ -791,6 +955,22 @@ public class Process {
}
this.rollbackType = rollbackType;
this.inventoryRollback = inventoryRollback;
this.retryRequired = retryRequired;
}
}
private static final class PendingEntitySpawnLog {
private final String user;
private final EntitySpawnData data;
private final boolean confirmIdentity;
private final boolean retryRequired;
private PendingEntitySpawnLog(String user, EntitySpawnData data, boolean confirmIdentity, boolean retryRequired) {
this.user = user;
this.data = data;
this.confirmIdentity = confirmIdentity;
this.retryRequired = retryRequired;
}
}
@@ -798,10 +978,12 @@ public class Process {
private final String user;
private final EntityInteraction interaction;
private final boolean retryRequired;
private PendingEntityInteraction(String user, EntityInteraction interaction) {
private PendingEntityInteraction(String user, EntityInteraction interaction, boolean retryRequired) {
this.user = user;
this.interaction = interaction;
this.retryRequired = retryRequired;
}
}
}
src/main/java/net/coreprotect/consumer/process/RollbackUpdateProcess.java
+15 -22
View File
@@ -1,7 +1,5 @@
package net.coreprotect.consumer.process;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
@@ -9,49 +7,44 @@ import java.util.Map;
import java.util.Map.Entry;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.database.Database;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.model.rollback.RollbackUpdateTargets;
import net.coreprotect.utility.MaterialUtils;
class RollbackUpdateProcess {
static void process(Statement statement, int processId, int id, int action, int table) {
process(statement, processId, id, action, table, RollbackUpdateTargets.usesInventoryRollbackState(table));
static void process(ConsumerWriteBatch batch, int processId, int id, int action, int table) throws Exception {
process(batch, processId, id, action, table, RollbackUpdateTargets.usesInventoryRollbackState(table));
}
static void process(Statement statement, int processId, int id, int action, int table, boolean inventoryRollback) {
static void process(ConsumerWriteBatch batch, int processId, int id, int action, int table, boolean inventoryRollback) throws Exception {
Map<Integer, List<Object[]>> updateLists = Consumer.consumerObjectArrayList.get(processId);
if (updateLists.get(id) != null) {
List<Object[]> list = updateLists.get(id);
process(statement, list, action, table, inventoryRollback);
updateLists.remove(id);
process(batch, list, action, table, inventoryRollback);
}
}
static void process(Statement statement, List<Object[]> list, int action, int table, boolean inventoryRollback) {
Map<Integer, List<Long>> rowIdsByValue = groupRowIdsByValue(list, action, inventoryRollback);
for (Entry<Integer, List<Long>> entry : rowIdsByValue.entrySet()) {
Database.performRolledBackUpdate(statement, entry.getKey(), entry.getValue(), table);
static void process(ConsumerWriteBatch batch, List<Object[]> list, int action, int table, boolean inventoryRollback) throws Exception {
Map<Integer, List<Object[]>> rowsByValue = groupRowsByValue(list, action, inventoryRollback);
for (Entry<Integer, List<Object[]>> entry : rowsByValue.entrySet()) {
batch.updateRolledBackRows(table, entry.getKey(), entry.getValue());
}
}
static void processChecked(Statement statement, List<Object[]> list, int action, int table, boolean inventoryRollback) throws SQLException {
Map<Integer, List<Long>> rowIdsByValue = groupRowIdsByValue(list, action, inventoryRollback);
for (Entry<Integer, List<Long>> entry : rowIdsByValue.entrySet()) {
Database.performRolledBackUpdateChecked(statement, entry.getKey(), entry.getValue(), table);
}
static void processChecked(ConsumerWriteBatch batch, List<Object[]> list, int action, int table, boolean inventoryRollback) throws Exception {
process(batch, list, action, table, inventoryRollback);
}
private static Map<Integer, List<Long>> groupRowIdsByValue(List<Object[]> list, int action, boolean inventoryRollback) {
Map<Integer, List<Long>> rowIdsByValue = new HashMap<>();
private static Map<Integer, List<Object[]>> groupRowsByValue(List<Object[]> list, int action, boolean inventoryRollback) {
Map<Integer, List<Object[]>> rowsByValue = new HashMap<>();
for (Object[] listRow : list) {
long rowid = (Long) listRow[0];
int rolledBack = (Integer) listRow[9];
if (MaterialUtils.rolledBack(rolledBack, inventoryRollback) == action) { // 1 = restore, 0 = rollback
int toggledValue = MaterialUtils.toggleRolledBack(rolledBack, inventoryRollback);
rowIdsByValue.computeIfAbsent(toggledValue, key -> new ArrayList<>()).add(rowid);
rowsByValue.computeIfAbsent(toggledValue, key -> new ArrayList<>()).add(listRow);
}
}
return rowIdsByValue;
return rowsByValue;
}
}
src/main/java/net/coreprotect/consumer/process/SignTextProcess.java
+2 -3
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.util.Map;
import org.bukkit.Location;
@@ -10,14 +10,13 @@ import net.coreprotect.database.logger.SignTextLogger;
class SignTextProcess {
static void process(PreparedStatement preparedStmt, int batchCount, int processId, int id, int forceData, String user, Object object, int action, int color) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, int processId, int id, int forceData, String user, Object object, int action, int color) {
if (object instanceof Location) {
Location location = (Location) object;
Map<Integer, Object[]> signs = Consumer.consumerSigns.get(processId);
if (signs.get(id) != null) {
Object[] SIGN_DATA = signs.get(id);
SignTextLogger.log(preparedStmt, batchCount, user, location, action, color, (Integer) SIGN_DATA[0], (Integer) SIGN_DATA[1], (Boolean) SIGN_DATA[2], (Boolean) SIGN_DATA[3], (String) SIGN_DATA[4], (String) SIGN_DATA[5], (String) SIGN_DATA[6], (String) SIGN_DATA[7], (String) SIGN_DATA[8], (String) SIGN_DATA[9], (String) SIGN_DATA[10], (String) SIGN_DATA[11], forceData);
signs.remove(id);
}
}
}
src/main/java/net/coreprotect/consumer/process/SignUpdateProcess.java
+2 -2
View File
@@ -27,10 +27,10 @@ class SignUpdateProcess {
int userid = ConfigHandler.playerIdCache.get(user.toLowerCase(Locale.ROOT));
String query = "";
if (action == 0) {
query = "SELECT color, color_secondary, data, waxed, face, line_1, line_2, line_3, line_4, line_5, line_6, line_7, line_8 FROM " + ConfigHandler.prefix + "sign WHERE user='" + userid + "' AND wid='" + wid + "' AND x='" + x + "' AND z='" + z + "' AND y='" + y + "' AND time < '" + time + "' ORDER BY rowid DESC LIMIT 0, 1";
query = "SELECT color, color_secondary, data, waxed, face, line_1, line_2, line_3, line_4, line_5, line_6, line_7, line_8 FROM " + ConfigHandler.prefix + "sign WHERE " + ConfigHandler.databaseType.getUserColumn() + "=" + userid + " AND wid=" + wid + " AND x=" + x + " AND z=" + z + " AND y=" + y + " AND time < " + time + " ORDER BY rowid DESC LIMIT 1 OFFSET 0";
}
else {
query = "SELECT color, color_secondary, data, waxed, face, line_1, line_2, line_3, line_4, line_5, line_6, line_7, line_8 FROM " + ConfigHandler.prefix + "sign WHERE user='" + userid + "' AND wid='" + wid + "' AND x='" + x + "' AND z='" + z + "' AND y='" + y + "' AND time >= '" + time + "' ORDER BY rowid ASC LIMIT 0, 1";
query = "SELECT color, color_secondary, data, waxed, face, line_1, line_2, line_3, line_4, line_5, line_6, line_7, line_8 FROM " + ConfigHandler.prefix + "sign WHERE " + ConfigHandler.databaseType.getUserColumn() + "=" + userid + " AND wid=" + wid + " AND x=" + x + " AND z=" + z + " AND y=" + y + " AND time >= " + time + " ORDER BY rowid ASC LIMIT 1 OFFSET 0";
}
SignStatement.getData(statement, block, query);
BlockUtils.updateBlock(block);
src/main/java/net/coreprotect/consumer/process/SkullUpdateProcess.java
+1 -1
View File
@@ -18,7 +18,7 @@ class SkullUpdateProcess {
*/
if (object instanceof BlockState) {
BlockState block = (BlockState) object;
String query = "SELECT owner, skin FROM " + ConfigHandler.prefix + "skull WHERE rowid='" + rowId + "' LIMIT 0, 1";
String query = "SELECT owner, skin FROM " + ConfigHandler.prefix + "skull WHERE rowid=" + rowId + " LIMIT 1 OFFSET 0";
SkullStatement.getData(statement, block, query);
BlockUtils.updateBlock(block);
}
src/main/java/net/coreprotect/consumer/process/StructureGrowthProcess.java
+2 -4
View File
@@ -1,6 +1,6 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import net.coreprotect.database.ConsumerWriteBatch;
import java.sql.Statement;
import java.util.List;
import java.util.Map;
@@ -12,11 +12,10 @@ import net.coreprotect.database.Lookup;
import net.coreprotect.database.logger.BlockBreakLogger;
import net.coreprotect.database.logger.BlockPlaceLogger;
import net.coreprotect.utility.MaterialUtils;
import net.coreprotect.utility.Util;
class StructureGrowthProcess {
static void process(Statement statement, PreparedStatement preparedStmt, int batchCount, int processId, int id, String user, Object object, int replaceBlockCount) {
static void process(Statement statement, ConsumerWriteBatch preparedStmt, int batchCount, int processId, int id, String user, Object object, int replaceBlockCount) {
if (object instanceof BlockState) {
BlockState block = (BlockState) object;
Map<Integer, List<BlockState>> blockLists = Consumer.consumerBlockList.get(processId);
@@ -39,7 +38,6 @@ class StructureGrowthProcess {
}
count++;
}
blockLists.remove(id);
}
}
}
src/main/java/net/coreprotect/consumer/process/WorldInsertProcess.java
+4 -4
View File
@@ -1,9 +1,9 @@
package net.coreprotect.consumer.process;
import java.sql.PreparedStatement;
import java.sql.Statement;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.MaterialStatement;
import net.coreprotect.database.statement.WorldStatement;
import net.coreprotect.language.Phrase;
@@ -12,10 +12,10 @@ import net.coreprotect.utility.Chat;
class WorldInsertProcess {
static void process(PreparedStatement preparedStmt, int batchCount, Statement statement, Object world, int worldId) {
static void process(ConsumerWriteBatch preparedStmt, int batchCount, Statement statement, Object world, int worldId) {
if (world instanceof String) {
String query = "SELECT id FROM " + ConfigHandler.prefix + "world WHERE id = '" + worldId + "' LIMIT 0, 1";
boolean hasMaterial = MaterialStatement.hasMaterial(statement, query);
String query = "SELECT id FROM " + ConfigHandler.prefix + "world WHERE id = " + worldId + " LIMIT 1 OFFSET 0";
boolean hasMaterial = !ConfigHandler.databaseType.isClickHouse() && MaterialStatement.hasMaterial(statement, query);
if (!hasMaterial) {
WorldStatement.insert(preparedStmt, batchCount, worldId, (String) world);
src/main/java/net/coreprotect/database/BlockLookup.java
+2 -6
View File
@@ -29,18 +29,14 @@ public class BlockLookup {
int z = block.getZ();
int time = (int) (System.currentTimeMillis() / 1000L);
int worldId = WorldUtils.getWorldId(block.getWorld().getName());
String query = "SELECT user,type FROM " + ConfigHandler.prefix + "block " + WorldUtils.getWidIndex("block") + "WHERE wid = '" + worldId + "' AND x = '" + x + "' AND z = '" + z + "' AND y = '" + y + "' AND rolled_back IN(0,2) AND action='1' ORDER BY rowid DESC LIMIT 0, 1";
String query = "SELECT " + ConfigHandler.databaseType.getUserColumn() + ",type FROM " + ConfigHandler.prefix + "block " + WorldUtils.getWidIndex("block") + "WHERE wid = " + worldId + " AND x = " + x + " AND z = " + z + " AND y = " + y + " AND rolled_back IN(0,2) AND action=1 ORDER BY rowid DESC LIMIT 1 OFFSET 0";
ResultSet results = statement.executeQuery(query);
while (results.next()) {
int resultUserId = results.getInt("user");
int resultType = results.getInt("type");
if (ConfigHandler.playerIdCacheReversed.get(resultUserId) == null) {
UserStatement.loadName(statement.getConnection(), resultUserId);
}
result = ConfigHandler.playerIdCacheReversed.get(resultUserId);
result = UserStatement.getName(statement.getConnection(), resultUserId);
if (result.length() > 0) {
Material resultMaterial = MaterialUtils.getType(resultType);
CacheHandler.lookupCache.put("" + x + "." + y + "." + z + "." + worldId + "", new Object[] { time, result, resultMaterial });
src/main/java/net/coreprotect/database/ConsumerEntitySpawnUpdates.java
+24
View File
@@ -0,0 +1,24 @@
package net.coreprotect.database;
import java.util.List;
import net.coreprotect.model.entity.EntityContainerRollbackUpdate;
import net.coreprotect.model.entity.EntitySpawnData;
public interface ConsumerEntitySpawnUpdates extends AutoCloseable {
default void prefetch(List<EntitySpawnData> updates) throws Exception {
}
void apply(EntitySpawnData data);
void applyCombined(EntityContainerRollbackUpdate update, Database.SavepointOperation rowUpdate);
void afterCommit(boolean committed);
void afterRetain();
@Override
void close() throws Exception;
}
src/main/java/net/coreprotect/database/ConsumerWriteBatch.java
+84
View File
@@ -0,0 +1,84 @@
package net.coreprotect.database;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
public interface ConsumerWriteBatch extends AutoCloseable {
enum ReferenceKind {
ART,
BLOCK_DATA,
ENTITY,
MATERIAL,
WORLD
}
void begin() throws Exception;
boolean commit() throws Exception;
default boolean shouldCommit() {
return false;
}
void rollback();
void executeAtomically(String name, Database.SavepointOperation operation) throws Exception;
int resolveUserId(String user, String uuid) throws Exception;
void recordUsername(String user, String uuid, int retainHistory, int time) throws Exception;
void updateDatabaseLock(int status, int time) throws Exception;
void addReference(ReferenceKind kind, int batchCount, int id, String value) throws Exception;
void addBlock(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws Exception;
long addBlockReturningId(int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws Exception;
int addSkull(int time, String owner, String skin) throws Exception;
void addContainer(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws Exception;
void addEntityContainer(int batchCount, int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws Exception;
void addItem(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, byte[] data, int amount, int action, int rolledBack) throws Exception;
void addChat(int batchCount, long time, int userId, int worldId, int x, int y, int z, String message) throws Exception;
void addCommand(int batchCount, long time, int userId, int worldId, int x, int y, int z, String message) throws Exception;
void addSession(int batchCount, int time, int userId, int worldId, int x, int y, int z, int action) throws Exception;
void addSign(int batchCount, int time, int userId, int worldId, int x, int y, int z, int action, int color, int colorSecondary, int data, int waxed, int face, String[] lines) throws Exception;
int addEntity(int time, byte[] data) throws Exception;
int addEntitySpawn(int time, Long blockRowId, Integer killRowId, UUID uuid, int originWorldId, int currentWorldId, double originX, double originY, double originZ, double currentX, double currentY, double currentZ, float yaw, float pitch, byte[] data, int removed) throws Exception;
void linkEntitySpawnBlock(int trackingRowId, long blockRowId) throws Exception;
void linkEntitySpawnKill(UUID uuid, int killRowId) throws Exception;
void checkpointEntitySpawn(int trackingRowId, int worldId, double x, double y, double z, float yaw, float pitch) throws Exception;
void addEntityInteraction(int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int action, byte[] metadata, int rolledBack) throws Exception;
void updateRolledBack(int target, int rolledBack, List<Long> rowIds) throws Exception;
default void updateRolledBackRows(int target, int rolledBack, List<Object[]> rows) throws Exception {
List<Long> rowIds = new ArrayList<>(rows.size());
for (Object[] row : rows) {
rowIds.add((Long) row[0]);
}
updateRolledBack(target, rolledBack, rowIds);
}
ConsumerEntitySpawnUpdates entitySpawnUpdates() throws Exception;
@Override
void close() throws Exception;
}
src/main/java/net/coreprotect/database/ContainerRollback.java
+29 -2
View File
@@ -20,6 +20,7 @@ import org.bukkit.inventory.ItemStack;
import net.coreprotect.CoreProtect;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.consumer.Queue;
import net.coreprotect.consumer.process.Process;
import net.coreprotect.database.rollback.Rollback;
@@ -36,14 +37,22 @@ import net.coreprotect.utility.ErrorReporter;
public class ContainerRollback extends Rollback {
public static void performContainerRollbackRestore(Statement statement, CommandSender user, List<String> checkUuids, List<String> checkUsers, String timeString, List<Object> restrictList, Map<Object, Boolean> excludeList, List<String> excludeUserList, List<Integer> actionList, final Location location, Integer[] radius, long startTime, long endTime, boolean restrictWorld, boolean lookup, boolean verbose, final int rollbackType) {
public static boolean performContainerRollbackRestore(Statement statement, CommandSender user, List<String> checkUuids, List<String> checkUsers, String timeString, List<Object> restrictList, Map<Object, Boolean> excludeList, List<String> excludeUserList, List<Integer> actionList, final Location location, Integer[] radius, long startTime, long endTime, boolean restrictWorld, boolean lookup, boolean verbose, final int rollbackType) {
try {
long timeStart = System.currentTimeMillis();
final List<Object[]> lookupList = Lookup.performLookupRaw(statement, user, checkUuids, checkUsers, restrictList, excludeList, excludeUserList, actionList, location, radius, null, startTime, endTime, -1, -1, restrictWorld, lookup);
if (lookupList == null) {
sendAborted(user);
return false;
}
if (rollbackType == 1) {
Collections.reverse(lookupList);
}
if (Consumer.isPersistenceHalted()) {
sendAborted(user);
return false;
}
String userString = "#server";
if (user != null) {
@@ -51,6 +60,10 @@ public class ContainerRollback extends Rollback {
}
Queue.queueRollbackUpdate(userString, location, lookupList, Process.CONTAINER_ROLLBACK_UPDATE, rollbackType); // Perform update transaction in consumer
if (Consumer.isPersistenceHalted()) {
sendAborted(user);
return false;
}
final String finalUserString = userString;
ConfigHandler.rollbackHash.put(userString, new int[] { 0, 0, 0, 0, 0 });
@@ -60,6 +73,10 @@ public class ContainerRollback extends Rollback {
public void run() {
try {
int[] rollbackHashData = ConfigHandler.rollbackHash.get(finalUserString);
if (Consumer.isPersistenceHalted()) {
ConfigHandler.rollbackHash.put(finalUserString, new int[] { rollbackHashData[0], rollbackHashData[1], rollbackHashData[2], 2, rollbackHashData[4] });
return;
}
int itemCount = rollbackHashData[0];
// int blockCount = rollbackHashData[1];
int entityCount = rollbackHashData[2];
@@ -154,6 +171,10 @@ public class ContainerRollback extends Rollback {
}
catch (Exception e) {
ErrorReporter.report(e);
int[] rollbackHashData = ConfigHandler.rollbackHash.get(finalUserString);
if (rollbackHashData != null) {
ConfigHandler.rollbackHash.put(finalUserString, new int[] { rollbackHashData[0], rollbackHashData[1], rollbackHashData[2], 2, rollbackHashData[4] });
}
}
}
}, location, 0);
@@ -172,7 +193,10 @@ public class ContainerRollback extends Rollback {
break;
}
}
if (next != 1) {
sendAborted(user);
return false;
}
rollbackHashData = ConfigHandler.rollbackHash.get(finalUserString);
int blockCount = rollbackHashData[1];
long timeFinish = System.currentTimeMillis();
@@ -188,9 +212,12 @@ public class ContainerRollback extends Rollback {
RollbackComplete.output(user, location, checkUsers, restrictList, excludeList, excludeUserList, actionList, timeString, file, totalSeconds, itemCount, blockCount, entityCount, rollbackType, radius, verbose, restrictWorld, 0);
}
return true;
}
catch (Exception e) {
ErrorReporter.report(e);
sendAborted(user);
return false;
}
}
src/main/java/net/coreprotect/database/Database.java
+418 -84
View File
@@ -8,10 +8,14 @@ import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.IdentityHashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import org.bukkit.Location;
@@ -23,6 +27,9 @@ import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.consumer.Queue;
import net.coreprotect.consumer.process.Process;
import net.coreprotect.database.clickhouse.ClickHouseConsumerWriteBatch;
import net.coreprotect.database.clickhouse.ClickHouseDatabase;
import net.coreprotect.database.clickhouse.ClickHouseJdbcConfig;
import net.coreprotect.language.Phrase;
import net.coreprotect.listener.player.InventoryChangeListener;
import net.coreprotect.model.BlockGroup;
@@ -31,6 +38,7 @@ import net.coreprotect.utility.Chat;
import net.coreprotect.utility.Color;
import net.coreprotect.utility.ItemUtils;
import net.coreprotect.utility.ErrorReporter;
import net.coreprotect.utility.VersionUtils;
public class Database extends Queue {
@@ -53,35 +61,55 @@ public class Database extends Queue {
public static final int ENTITY_INTERACTION = 16;
private static final int ROLLED_BACK_UPDATE_BATCH_SIZE = 1000;
private static final int DUCKDB_ROLLED_BACK_UPDATE_BATCH_SIZE = 5000;
private static final long CLICKHOUSE_CONNECTION_ERROR_INTERVAL_NANOS = 30_000_000_000L;
private static final Map<Integer, String> SQL_QUERIES = new HashMap<>();
private static final Set<Connection> ACTIVE_CONNECTIONS = Collections.newSetFromMap(new IdentityHashMap<>());
private static final ThreadLocal<Boolean> TRANSACTION_ROLLBACK_ONLY = ThreadLocal.withInitial(() -> false);
private static final ThreadLocal<Boolean> TRANSACTION_ROLLBACK_ACKNOWLEDGED = ThreadLocal.withInitial(() -> false);
private static volatile ClickHouseDatabase clickHouseDatabase;
private static long nextClickHouseConnectionErrorReport;
static {
// Initialize SQL queries for different table types
SQL_QUERIES.put(SIGN, "INSERT INTO %sprefix%sign (time, user, wid, x, y, z, action, color, color_secondary, data, waxed, face, line_1, line_2, line_3, line_4, line_5, line_6, line_7, line_8) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(BLOCK, "INSERT INTO %sprefix%block (time, user, wid, x, y, z, type, data, meta, blockdata, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(SIGN, "INSERT INTO %sprefix%sign (time, %user%, wid, x, y, z, action, color, color_secondary, data, waxed, face, line_1, line_2, line_3, line_4, line_5, line_6, line_7, line_8) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(BLOCK, "INSERT INTO %sprefix%block (time, %user%, wid, x, y, z, type, data, meta, blockdata, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(SKULL, "INSERT INTO %sprefix%skull (time, owner, skin) VALUES (?, ?, ?)");
SQL_QUERIES.put(CONTAINER, "INSERT INTO %sprefix%container (time, user, wid, x, y, z, type, data, amount, metadata, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(ITEM, "INSERT INTO %sprefix%item (time, user, wid, x, y, z, type, data, amount, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(CONTAINER, "INSERT INTO %sprefix%container (time, %user%, wid, x, y, z, type, data, amount, metadata, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(ITEM, "INSERT INTO %sprefix%item (time, %user%, wid, x, y, z, type, data, amount, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(WORLD, "INSERT INTO %sprefix%world (id, world) VALUES (?, ?)");
SQL_QUERIES.put(CHAT, "INSERT INTO %sprefix%chat (time, user, wid, x, y, z, message) VALUES (?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(COMMAND, "INSERT INTO %sprefix%command (time, user, wid, x, y, z, message) VALUES (?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(SESSION, "INSERT INTO %sprefix%session (time, user, wid, x, y, z, action) VALUES (?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(CHAT, "INSERT INTO %sprefix%chat (time, %user%, wid, x, y, z, message) VALUES (?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(COMMAND, "INSERT INTO %sprefix%command (time, %user%, wid, x, y, z, message) VALUES (?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(SESSION, "INSERT INTO %sprefix%session (time, %user%, wid, x, y, z, action) VALUES (?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(ENTITY, "INSERT INTO %sprefix%entity (time, data) VALUES (?, ?)");
SQL_QUERIES.put(MATERIAL, "INSERT INTO %sprefix%material_map (id, material) VALUES (?, ?)");
SQL_QUERIES.put(ART, "INSERT INTO %sprefix%art_map (id, art) VALUES (?, ?)");
SQL_QUERIES.put(ENTITY_MAP, "INSERT INTO %sprefix%entity_map (id, entity) VALUES (?, ?)");
SQL_QUERIES.put(BLOCKDATA, "INSERT INTO %sprefix%blockdata_map (id, data) VALUES (?, ?)");
SQL_QUERIES.put(ENTITY_SPAWN, "INSERT INTO %sprefix%entity_spawn (time, uuid, wid, current_wid, origin_x, origin_y, origin_z, x, y, z, yaw, pitch, data, removed) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(ENTITY_CONTAINER, "INSERT INTO %sprefix%entity_container (time, user, entity_spawn_rowid, wid, x, y, z, type, data, amount, metadata, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(ENTITY_INTERACTION, "INSERT INTO %sprefix%entity_interaction (time, user, entity_spawn_rowid, wid, x, y, z, type, action, metadata, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(ENTITY_CONTAINER, "INSERT INTO %sprefix%entity_container (time, %user%, entity_spawn_rowid, wid, x, y, z, type, data, amount, metadata, action, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
SQL_QUERIES.put(ENTITY_INTERACTION, "INSERT INTO %sprefix%entity_interaction (time, %user%, entity_spawn_rowid, wid, x, y, z, type, action, metadata, rolled_back) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
}
public static void beginTransaction(Statement statement, boolean isMySQL) throws SQLException {
beginTransaction(statement, isMySQL ? DatabaseType.MYSQL : DatabaseType.SQLITE);
}
public static void beginTransaction(Statement statement, DatabaseType databaseType) throws SQLException {
rejectClickHouseTransaction(databaseType);
Consumer.transacting = true;
TRANSACTION_ROLLBACK_ONLY.set(false);
TRANSACTION_ROLLBACK_ACKNOWLEDGED.set(false);
boolean started = false;
try {
if (isMySQL) {
if (databaseType.isDuckDB()) {
statement.getConnection().setAutoCommit(false);
try (ResultSet ignored = statement.executeQuery("SELECT 1")) {
// DuckDB starts JDBC transactions lazily.
}
}
else if (databaseType.isMySQL()) {
statement.executeUpdate("START TRANSACTION");
}
else {
@@ -93,27 +121,56 @@ public class Database extends Queue {
if (!started) {
Consumer.transacting = false;
Consumer.interrupt = false;
TRANSACTION_ROLLBACK_ONLY.remove();
TRANSACTION_ROLLBACK_ACKNOWLEDGED.remove();
}
}
}
public static void commitTransaction(Statement statement, boolean isMySQL) throws Exception {
commitTransactionChecked(statement, isMySQL);
commitTransactionChecked(statement, isMySQL ? DatabaseType.MYSQL : DatabaseType.SQLITE);
}
public static boolean commitTransactionChecked(Statement statement, boolean isMySQL) throws Exception {
return commitTransactionChecked(statement, isMySQL ? DatabaseType.MYSQL : DatabaseType.SQLITE);
}
public static boolean commitTransactionChecked(Statement statement, DatabaseType databaseType) throws Exception {
rejectClickHouseTransaction(databaseType);
if (TRANSACTION_ROLLBACK_ONLY.get()) {
if (databaseType.isDuckDB() && TRANSACTION_ROLLBACK_ACKNOWLEDGED.get()) {
return rollbackTransaction(statement, databaseType);
}
return false;
}
int count = 0;
while (true) {
try {
if (isMySQL) {
statement.executeUpdate("COMMIT");
if (databaseType.isDuckDB()) {
Connection connection = statement.getConnection();
connection.commit();
try {
connection.setAutoCommit(true);
}
catch (Exception cleanupException) {
ErrorReporter.report(cleanupException);
try {
connection.close();
}
catch (Exception closeException) {
ErrorReporter.report(closeException);
}
}
}
else {
statement.executeUpdate("COMMIT TRANSACTION");
statement.executeUpdate(databaseType.isMySQL() ? "COMMIT" : "COMMIT TRANSACTION");
}
Consumer.transacting = false;
Consumer.interrupt = false;
TRANSACTION_ROLLBACK_ONLY.remove();
TRANSACTION_ROLLBACK_ACKNOWLEDGED.remove();
return true;
}
catch (Exception e) {
@@ -126,14 +183,44 @@ public class Database extends Queue {
ErrorReporter.report(e);
Consumer.transacting = false;
Consumer.interrupt = false;
TRANSACTION_ROLLBACK_ONLY.remove();
TRANSACTION_ROLLBACK_ACKNOWLEDGED.remove();
return false;
}
}
}
public static void rollbackTransaction(Statement statement, boolean isMySQL) {
public static boolean rollbackTransaction(Statement statement, boolean isMySQL) {
return rollbackTransaction(statement, isMySQL ? DatabaseType.MYSQL : DatabaseType.SQLITE);
}
public static boolean rollbackTransaction(Statement statement, DatabaseType databaseType) {
if (databaseType.isClickHouse()) {
throw new UnsupportedOperationException("ClickHouse writes use immutable consumer batches, not JDBC transactions");
}
boolean rolledBack = false;
try {
statement.executeUpdate(isMySQL ? "ROLLBACK" : "ROLLBACK TRANSACTION");
if (databaseType.isDuckDB()) {
Connection connection = statement.getConnection();
connection.rollback();
rolledBack = true;
try {
connection.setAutoCommit(true);
}
catch (Exception cleanupException) {
ErrorReporter.report(cleanupException);
try {
connection.close();
}
catch (Exception closeException) {
ErrorReporter.report(closeException);
}
}
}
else {
statement.executeUpdate(databaseType.isMySQL() ? "ROLLBACK" : "ROLLBACK TRANSACTION");
rolledBack = true;
}
}
catch (Exception e) {
ErrorReporter.report(e);
@@ -141,10 +228,39 @@ public class Database extends Queue {
finally {
Consumer.transacting = false;
Consumer.interrupt = false;
TRANSACTION_ROLLBACK_ONLY.remove();
TRANSACTION_ROLLBACK_ACKNOWLEDGED.remove();
}
return rolledBack;
}
public static void acknowledgeRollbackOnlyTransaction() {
if (TRANSACTION_ROLLBACK_ONLY.get()) {
TRANSACTION_ROLLBACK_ACKNOWLEDGED.set(true);
}
}
static boolean isTransactionRollbackOnly() {
return TRANSACTION_ROLLBACK_ONLY.get();
}
static boolean isRollbackOnlyTransactionAcknowledged() {
return TRANSACTION_ROLLBACK_ONLY.get() && TRANSACTION_ROLLBACK_ACKNOWLEDGED.get();
}
public static void executeSavepoint(Statement statement, String name, SavepointOperation operation) throws Exception {
rejectClickHouseTransaction(ConfigHandler.databaseType);
if (ConfigHandler.databaseType.isDuckDB()) {
try {
operation.execute();
}
catch (Exception e) {
TRANSACTION_ROLLBACK_ONLY.set(true);
throw e;
}
return;
}
statement.execute("SAVEPOINT " + name);
try {
operation.execute();
@@ -155,12 +271,14 @@ public class Database extends Queue {
statement.execute("ROLLBACK TO SAVEPOINT " + name);
}
catch (Exception rollbackException) {
TRANSACTION_ROLLBACK_ONLY.set(true);
e.addSuppressed(rollbackException);
}
try {
statement.execute("RELEASE SAVEPOINT " + name);
}
catch (Exception releaseException) {
TRANSACTION_ROLLBACK_ONLY.set(true);
e.addSuppressed(releaseException);
}
throw e;
@@ -173,6 +291,12 @@ public class Database extends Queue {
}
}
public static void performCheckpoint(Statement statement, DatabaseType databaseType) throws SQLException {
if (databaseType.isSQLite()) {
statement.executeUpdate("PRAGMA wal_checkpoint(TRUNCATE)");
}
}
public static void setMultiInt(PreparedStatement statement, int value, int count) {
try {
for (int i = 1; i <= count; i++) {
@@ -185,7 +309,17 @@ public class Database extends Queue {
}
public static boolean hasReturningKeys() {
return (!Config.getGlobal().MYSQL && ConfigHandler.SERVER_VERSION >= 20);
return ConfigHandler.databaseType.isDuckDB() || (ConfigHandler.databaseType.isSQLite() && ConfigHandler.SERVER_VERSION >= 20);
}
public static void handleWriteFailure(Exception exception) {
if (ConfigHandler.databaseType.isColumnar()) {
if (exception instanceof DatabaseWriteException) {
throw (DatabaseWriteException) exception;
}
throw new DatabaseWriteException(exception);
}
ErrorReporter.report(exception);
}
public static void containerBreakCheck(String user, Material type, Object container, ItemStack[] contents, Location location) {
@@ -222,11 +356,18 @@ public class Database extends Queue {
public static Connection getConnection(boolean force, boolean startup, boolean onlyCheckTransacting, int waitTime) {
Connection connection = null;
if (Consumer.isDatabaseReloadBlocked()) {
return null;
}
Consumer.lockDatabaseAccess();
try {
if (Consumer.isDatabaseReloadBlocked()) {
return null;
}
if (!force && (ConfigHandler.converterRunning || ConfigHandler.purgeRunning)) {
return connection;
}
if (Config.getGlobal().MYSQL) {
if (ConfigHandler.databaseType.isMySQL()) {
try {
connection = ConfigHandler.hikariDataSource.getConnection();
ConfigHandler.databaseReachable = true;
@@ -237,6 +378,14 @@ public class Database extends Queue {
ErrorReporter.report(e);
}
}
else if (ConfigHandler.databaseType.isDuckDB()) {
connection = DuckDBDatabase.getConnection();
ConfigHandler.databaseReachable = true;
}
else if (ConfigHandler.databaseType.isClickHouse()) {
connection = requireClickHouseDatabase().openConnection();
ConfigHandler.databaseReachable = true;
}
else {
if (Consumer.transacting && onlyCheckTransacting) {
Consumer.interrupt = true;
@@ -257,24 +406,151 @@ public class Database extends Queue {
ConfigHandler.databaseReachable = true;
}
if (connection != null) {
registerConnection(connection);
}
}
catch (Exception e) {
ErrorReporter.report(e);
if (ConfigHandler.databaseType.isColumnar()) {
ConfigHandler.databaseReachable = false;
}
if (!ConfigHandler.databaseType.isClickHouse() || shouldReportClickHouseConnectionError()) {
ErrorReporter.report(e);
}
}
finally {
Consumer.unlockDatabaseAccess();
}
return connection;
}
public static boolean awaitConnectionDrain(long timeoutMillis) throws InterruptedException {
long deadline = System.nanoTime() + timeoutMillis * 1_000_000L;
while (hasActiveConnections()) {
if (System.nanoTime() >= deadline) {
return false;
}
Thread.sleep(10L);
}
return true;
}
private static void registerConnection(Connection connection) {
synchronized (ACTIVE_CONNECTIONS) {
removeClosedConnections(false);
ACTIVE_CONNECTIONS.add(connection);
}
}
private static boolean hasActiveConnections() {
synchronized (ACTIVE_CONNECTIONS) {
removeClosedConnections(true);
return !ACTIVE_CONNECTIONS.isEmpty();
}
}
private static void removeClosedConnections(boolean draining) {
Iterator<Connection> iterator = ACTIVE_CONNECTIONS.iterator();
while (iterator.hasNext()) {
Connection connection = iterator.next();
try {
if (connection.isClosed()) {
iterator.remove();
}
}
catch (SQLException exception) {
if (draining) {
try {
connection.close();
}
catch (SQLException closeException) {
exception.addSuppressed(closeException);
}
iterator.remove();
ErrorReporter.report(exception);
}
}
}
}
private static synchronized boolean shouldReportClickHouseConnectionError() {
long now = System.nanoTime();
if (now < nextClickHouseConnectionErrorReport) {
return false;
}
nextClickHouseConnectionErrorReport = now + CLICKHOUSE_CONNECTION_ERROR_INTERVAL_NANOS;
return true;
}
public static void closeConnection() {
try {
if (ConfigHandler.hikariDataSource != null) {
if (ConfigHandler.hikariDataSource != null) {
try {
ConfigHandler.hikariDataSource.close();
}
catch (Exception e) {
ErrorReporter.report(e);
}
finally {
ConfigHandler.hikariDataSource = null;
}
}
try {
DuckDBDatabase.close();
}
catch (Exception e) {
ErrorReporter.report(e);
}
closeClickHouse();
}
public static synchronized void initializeClickHouse(ClickHouseJdbcConfig config, String prefix) throws SQLException {
closeClickHouseChecked();
ClickHouseDatabase initialized = ClickHouseDatabase.initialize(config, prefix);
try {
Integer[] version = VersionUtils.getInternalPluginVersion();
initialized.ensureCoreData(version[0] + "." + version[1] + "." + version[2]);
}
catch (SQLException | RuntimeException exception) {
try {
initialized.close();
}
catch (SQLException closeException) {
exception.addSuppressed(closeException);
}
throw exception;
}
clickHouseDatabase = initialized;
ConfigHandler.databaseTables.clear();
ConfigHandler.databaseTables.addAll(DATABASE_TABLES);
}
public static ConsumerWriteBatch openConsumerWriteBatch(Connection connection) throws SQLException {
if (ConfigHandler.databaseType.isClickHouse()) {
return new ClickHouseConsumerWriteBatch(requireClickHouseDatabase(), ConfigHandler.prefix);
}
return new RelationalConsumerWriteBatch(connection, ConfigHandler.databaseType);
}
public static void recordDatabaseVersion(Statement statement, String version) throws SQLException {
if (ConfigHandler.databaseType.isClickHouse()) {
requireClickHouseDatabase().updateCoreVersion(version);
return;
}
int timestamp = (int) (System.currentTimeMillis() / 1000L);
statement.executeUpdate("INSERT INTO " + ConfigHandler.prefix + "version (time,version) VALUES ('" + timestamp + "', '" + version + "')");
}
public static long purgeClickHouse(long startTime, long endTime, int worldId, List<Integer> blockTypes, boolean optimize) throws SQLException {
return requireClickHouseDatabase().purge(startTime, endTime, worldId, blockTypes, optimize);
}
public static void cancelClickHousePurge() {
ClickHouseDatabase database = clickHouseDatabase;
if (database != null) {
database.cancelPurge();
}
}
public static void performRolledBackUpdate(Statement statement, int rolledBack, List<Long> rowIds, int table) {
@@ -282,8 +558,9 @@ public class Database extends Queue {
try {
int listSize = rowIds.size();
for (int startIndex = 0; startIndex < listSize; startIndex += ROLLED_BACK_UPDATE_BATCH_SIZE) {
int endIndex = Math.min(startIndex + ROLLED_BACK_UPDATE_BATCH_SIZE, listSize);
int batchSize = getRolledBackUpdateBatchSize();
for (int startIndex = 0; startIndex < listSize; startIndex += batchSize) {
int endIndex = Math.min(startIndex + batchSize, listSize);
StringBuilder query = new StringBuilder("UPDATE " + ConfigHandler.prefix + tableName + " SET rolled_back='" + rolledBack + "' WHERE rowid IN(");
for (int index = startIndex; index < endIndex; index++) {
if (index > startIndex) {
@@ -296,15 +573,16 @@ public class Database extends Queue {
}
}
catch (Exception e) {
ErrorReporter.report(e);
handleWriteFailure(e);
}
}
public static void performRolledBackUpdateChecked(Statement statement, int rolledBack, List<Long> rowIds, int table) throws SQLException {
String tableName = getRolledBackTableName(table);
int listSize = rowIds.size();
for (int startIndex = 0; startIndex < listSize; startIndex += ROLLED_BACK_UPDATE_BATCH_SIZE) {
int endIndex = Math.min(startIndex + ROLLED_BACK_UPDATE_BATCH_SIZE, listSize);
int batchSize = getRolledBackUpdateBatchSize();
for (int startIndex = 0; startIndex < listSize; startIndex += batchSize) {
int endIndex = Math.min(startIndex + batchSize, listSize);
StringBuilder rowIdsSql = new StringBuilder();
for (int index = startIndex; index < endIndex; index++) {
if (index > startIndex) {
@@ -329,6 +607,10 @@ public class Database extends Queue {
}
}
private static int getRolledBackUpdateBatchSize() {
return ConfigHandler.databaseType.isDuckDB() ? DUCKDB_ROLLED_BACK_UPDATE_BATCH_SIZE : ROLLED_BACK_UPDATE_BATCH_SIZE;
}
private static String getRolledBackTableName(int table) {
if (table == RollbackUpdateTargets.ENTITY_CONTAINER) {
return "entity_container";
@@ -348,6 +630,7 @@ public class Database extends Queue {
String query = SQL_QUERIES.get(type);
if (query != null) {
query = query.replace("%sprefix%", ConfigHandler.prefix);
query = query.replace("%user%", ConfigHandler.databaseType.getUserColumn());
preparedStatement = prepareStatement(connection, query, keys);
}
}
@@ -380,69 +663,77 @@ public class Database extends Queue {
return preparedStatement;
}
private static void initializeTables(String prefix, Statement statement) {
try {
if (!Config.getGlobal().MYSQL) {
if (!Config.getGlobal().DISABLE_WAL) {
statement.executeUpdate("PRAGMA journal_mode=WAL;");
}
else {
statement.executeUpdate("PRAGMA journal_mode=DELETE;");
}
private static void initializeTables(String prefix, Statement statement, DatabaseType databaseType) throws SQLException {
if (databaseType.isSQLite()) {
if (!Config.getGlobal().DISABLE_WAL) {
statement.executeUpdate("PRAGMA journal_mode=WAL;");
}
boolean lockInitialized = false;
String query = "SELECT rowid as id FROM " + prefix + "database_lock WHERE rowid='1' LIMIT 1";
ResultSet rs = statement.executeQuery(query);
while (rs.next()) {
lockInitialized = true;
}
rs.close();
if (!lockInitialized) {
int unixtimestamp = (int) (System.currentTimeMillis() / 1000L);
statement.executeUpdate("INSERT INTO " + prefix + "database_lock (rowid, status, time) VALUES ('1', '0', '" + unixtimestamp + "')");
Process.lastLockUpdate = 0;
else {
statement.executeUpdate("PRAGMA journal_mode=DELETE;");
}
}
catch (Exception e) {
ErrorReporter.report(e);
boolean lockInitialized = false;
String query = "SELECT rowid as id FROM " + prefix + "database_lock WHERE rowid='1' LIMIT 1";
try (ResultSet resultSet = statement.executeQuery(query)) {
lockInitialized = resultSet.next();
}
if (!lockInitialized) {
int unixtimestamp = (int) (System.currentTimeMillis() / 1000L);
statement.executeUpdate("INSERT INTO " + prefix + "database_lock (rowid, status, time) VALUES ('1', '0', '" + unixtimestamp + "')");
Process.lastLockUpdate = 0;
}
}
private static final List<String> DATABASE_TABLES = Arrays.asList("art_map", "block", "chat", "command", "container", "entity_container", "entity_interaction", "item", "database_lock", "entity", "entity_spawn", "entity_map", "material_map", "blockdata_map", "session", "sign", "skull", "user", "username_log", "version", "world");
static final List<String> DATABASE_TABLES = Arrays.asList("art_map", "block", "chat", "command", "container", "entity_container", "entity_interaction", "item", "database_lock", "entity", "entity_spawn", "entity_map", "material_map", "blockdata_map", "session", "sign", "skull", "user", "username_log", "version", "world");
public static void createDatabaseTables(String prefix, boolean forcePrefix, Connection forceConnection, boolean mySQL, boolean purge) {
createDatabaseTables(prefix, forcePrefix, forceConnection, mySQL ? DatabaseType.MYSQL : DatabaseType.SQLITE, purge);
}
public static void createDatabaseTables(String prefix, boolean forcePrefix, Connection forceConnection, DatabaseType databaseType, boolean purge) {
if (databaseType.isClickHouse()) {
throw new UnsupportedOperationException("ClickHouse schema creation is managed by ClickHouseDatabase");
}
ConfigHandler.databaseTables.clear();
ConfigHandler.databaseTables.addAll(DATABASE_TABLES);
if (mySQL) {
if (databaseType.isMySQL()) {
createMySQLTables(prefix, forceConnection, purge);
}
else if (databaseType.isDuckDB()) {
try {
DuckDBDatabase.createTables(prefix, forceConnection, purge);
}
catch (Exception e) {
ConfigHandler.databaseReachable = false;
throw new IllegalStateException("Failed to initialize the DuckDB schema", e);
}
}
else {
createSQLiteTables(prefix, forcePrefix, forceConnection, purge);
}
}
private static void createMySQLTables(String prefix, Connection forceConnection, boolean purge) {
boolean success = false;
try (Connection connection = (forceConnection != null ? forceConnection : Database.getConnection(true, true, true, 0))) {
if (connection != null) {
Statement statement = connection.createStatement();
if (connection == null) {
throw new SQLException("Unable to connect to MySQL while initializing the schema");
}
try (Statement statement = connection.createStatement()) {
createMySQLTableStructures(prefix, statement);
createMySQLIndexes(prefix, statement, purge);
if (!purge && forceConnection == null) {
initializeTables(prefix, statement);
initializeTables(prefix, statement, DatabaseType.MYSQL);
}
statement.close();
success = true;
}
}
catch (Exception e) {
ErrorReporter.report(e);
}
if (!success && forceConnection == null) {
Config.getGlobal().MYSQL = false;
if (forceConnection == null) {
Config.getGlobal().MYSQL = false;
}
throw new IllegalStateException("Failed to initialize the MySQL schema", e);
}
}
@@ -638,34 +929,37 @@ public class Database extends Queue {
private static void createSQLiteTables(String prefix, boolean forcePrefix, Connection forceConnection, boolean purge) {
try (Connection connection = (forceConnection != null ? forceConnection : Database.getConnection(true, 0))) {
Statement statement = connection.createStatement();
List<String> tableData = new ArrayList<>();
List<String> indexData = new ArrayList<>();
String attachDatabase = "";
if (purge && forceConnection == null) {
String query = "ATTACH DATABASE '" + ConfigHandler.path + ConfigHandler.sqlite + ".tmp' AS tmp_db";
PreparedStatement preparedStmt = connection.prepareStatement(query);
preparedStmt.execute();
preparedStmt.close();
attachDatabase = "tmp_db.";
if (connection == null) {
throw new SQLException("Unable to open SQLite while initializing the schema");
}
try (Statement statement = connection.createStatement()) {
List<String> tableData = new ArrayList<>();
List<String> indexData = new ArrayList<>();
String attachDatabase = "";
identifyExistingTablesAndIndexes(statement, attachDatabase, tableData, indexData);
String tablePrefix = forcePrefix ? prefix : ConfigHandler.prefix;
boolean createChatMessagePrefixIndex = !tableData.contains(tablePrefix + "chat");
boolean createCommandMessagePrefixIndex = !tableData.contains(tablePrefix + "command");
boolean createSignMessagePrefixIndexes = !tableData.contains(tablePrefix + "sign");
createSQLiteTableStructures(prefix, statement, tableData);
createSQLiteIndexes(tablePrefix, statement, indexData, attachDatabase, purge, createChatMessagePrefixIndex, createCommandMessagePrefixIndex, createSignMessagePrefixIndexes);
if (purge && forceConnection == null) {
String query = "ATTACH DATABASE '" + ConfigHandler.path + ConfigHandler.sqlite + ".tmp' AS tmp_db";
try (PreparedStatement preparedStatement = connection.prepareStatement(query)) {
preparedStatement.execute();
}
attachDatabase = "tmp_db.";
}
if (!purge && forceConnection == null) {
initializeTables(prefix, statement);
identifyExistingTablesAndIndexes(statement, attachDatabase, tableData, indexData);
String tablePrefix = forcePrefix ? prefix : ConfigHandler.prefix;
boolean createChatMessagePrefixIndex = !tableData.contains(tablePrefix + "chat");
boolean createCommandMessagePrefixIndex = !tableData.contains(tablePrefix + "command");
boolean createSignMessagePrefixIndexes = !tableData.contains(tablePrefix + "sign");
createSQLiteTableStructures(prefix, statement, tableData);
createSQLiteIndexes(tablePrefix, statement, indexData, attachDatabase, purge, createChatMessagePrefixIndex, createCommandMessagePrefixIndex, createSignMessagePrefixIndexes);
if (!purge && forceConnection == null) {
initializeTables(prefix, statement, DatabaseType.SQLITE);
}
}
statement.close();
}
catch (Exception e) {
ErrorReporter.report(e);
throw new IllegalStateException("Failed to initialize the SQLite schema", e);
}
}
@@ -831,4 +1125,44 @@ public class Database extends Queue {
void execute() throws Exception;
}
private static ClickHouseDatabase requireClickHouseDatabase() throws SQLException {
ClickHouseDatabase database = clickHouseDatabase;
if (database == null) {
throw new SQLException("ClickHouse is not initialized");
}
return database;
}
private static void closeClickHouse() {
try {
closeClickHouseChecked();
}
catch (SQLException exception) {
ErrorReporter.report(exception);
}
}
private static synchronized void closeClickHouseChecked() throws SQLException {
ClickHouseDatabase database = clickHouseDatabase;
clickHouseDatabase = null;
if (database != null) {
database.close();
}
}
private static void rejectClickHouseTransaction(DatabaseType databaseType) throws SQLException {
if (databaseType.isClickHouse()) {
throw new SQLException("ClickHouse writes use immutable consumer batches, not JDBC transactions");
}
}
private static final class DatabaseWriteException extends IllegalStateException {
private static final long serialVersionUID = 1L;
private DatabaseWriteException(Exception cause) {
super("Unable to map or bind database write data", cause);
}
}
}
src/main/java/net/coreprotect/database/DatabaseType.java
+70
View File
@@ -0,0 +1,70 @@
package net.coreprotect.database;
import java.util.Locale;
public enum DatabaseType {
CLICKHOUSE("ClickHouse"),
DUCKDB("DuckDB"),
MYSQL("MySQL"),
SQLITE("SQLite");
private final String displayName;
DatabaseType(String displayName) {
this.displayName = displayName;
}
public String getDisplayName() {
return displayName;
}
public String getUserColumn() {
if (isDuckDB()) {
return "\"user\"";
}
return isClickHouse() ? "`user`" : "user";
}
public boolean isDuckDB() {
return this == DUCKDB;
}
public boolean isClickHouse() {
return this == CLICKHOUSE;
}
public boolean isColumnar() {
return isClickHouse() || isDuckDB();
}
public boolean isEmbedded() {
return isDuckDB() || isSQLite();
}
public boolean isMySQL() {
return this == MYSQL;
}
public boolean isSQLite() {
return this == SQLITE;
}
public static DatabaseType parse(String value, boolean legacyMySQL) {
if (value == null) {
return legacyMySQL ? MYSQL : SQLITE;
}
String normalized = value.trim();
if (normalized.isEmpty()) {
throw new IllegalArgumentException("Database engine cannot be empty");
}
try {
return valueOf(normalized.toUpperCase(Locale.ROOT));
}
catch (IllegalArgumentException exception) {
throw new IllegalArgumentException("Unsupported database engine: " + value, exception);
}
}
}
src/main/java/net/coreprotect/database/DuckDBDatabase.java
+122
View File
@@ -0,0 +1,122 @@
package net.coreprotect.database;
import java.io.File;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.Properties;
import org.duckdb.DuckDBConnection;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.process.Process;
final class DuckDBDatabase {
private static DuckDBConnection rootConnection;
private DuckDBDatabase() {
throw new IllegalStateException("Database class");
}
static synchronized void open() throws Exception {
if (rootConnection != null && !rootConnection.isClosed()) {
return;
}
Class.forName("org.duckdb.DuckDBDriver");
File databaseFile = new File(ConfigHandler.path, ConfigHandler.duckdb);
File parent = databaseFile.getParentFile();
if (parent != null) {
parent.mkdirs();
}
Properties properties = new Properties();
properties.setProperty("memory_limit", ConfigHandler.duckdbMemoryLimit);
properties.setProperty("threads", Integer.toString(ConfigHandler.duckdbThreads));
properties.setProperty("temp_directory", databaseFile.getAbsolutePath() + ".tmp");
properties.setProperty("max_temp_directory_size", ConfigHandler.duckdbMaxTempDirectorySize);
properties.setProperty("enable_external_access", "false");
properties.setProperty("allow_unsigned_extensions", "false");
properties.setProperty("allow_community_extensions", "false");
properties.setProperty("autoload_known_extensions", "false");
properties.setProperty("autoinstall_known_extensions", "false");
rootConnection = (DuckDBConnection) java.sql.DriverManager.getConnection("jdbc:duckdb:" + databaseFile.getAbsolutePath(), properties);
}
static synchronized Connection getConnection() throws Exception {
open();
return rootConnection.duplicate();
}
static synchronized void close() throws Exception {
DuckDBConnection connection = rootConnection;
rootConnection = null;
if (connection != null) {
connection.close();
}
}
static void createTables(String prefix, Connection forceConnection, boolean purge) throws Exception {
Connection connection = forceConnection != null ? forceConnection : getConnection();
try {
try (Statement statement = connection.createStatement()) {
for (String table : Database.DATABASE_TABLES) {
if (!table.equals("database_lock")) {
statement.executeUpdate("CREATE SEQUENCE IF NOT EXISTS " + sequence(prefix, table) + " START 1");
}
}
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "art_map (" + rowId(prefix, "art_map") + ", id INTEGER, art VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "block (" + rowId(prefix, "block") + ", time INTEGER, \"user\" INTEGER, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, type INTEGER, data INTEGER, meta BLOB, blockdata BLOB, action TINYINT, rolled_back TINYINT)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "chat (" + rowId(prefix, "chat") + ", time INTEGER, \"user\" INTEGER, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, message VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "command (" + rowId(prefix, "command") + ", time INTEGER, \"user\" INTEGER, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, message VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "container (" + rowId(prefix, "container") + ", time INTEGER, \"user\" INTEGER, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, type INTEGER, data INTEGER, amount INTEGER, metadata BLOB, action TINYINT, rolled_back TINYINT)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "entity_container (" + rowId(prefix, "entity_container") + ", time INTEGER, \"user\" INTEGER, entity_spawn_rowid INTEGER NOT NULL, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, type INTEGER, data INTEGER, amount INTEGER, metadata BLOB, action TINYINT, rolled_back TINYINT)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "entity_interaction (" + rowId(prefix, "entity_interaction") + ", time INTEGER, \"user\" INTEGER, entity_spawn_rowid INTEGER NOT NULL, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, type INTEGER, action TINYINT, metadata BLOB, rolled_back TINYINT)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "item (" + rowId(prefix, "item") + ", time INTEGER, \"user\" INTEGER, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, type INTEGER, data BLOB, amount INTEGER, action TINYINT, rolled_back TINYINT)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "database_lock (rowid INTEGER PRIMARY KEY, status TINYINT, time INTEGER)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "entity (" + rowId(prefix, "entity") + ", time INTEGER, data BLOB)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "entity_spawn (" + rowId(prefix, "entity_spawn") + ", time INTEGER, block_rowid BIGINT, kill_rowid INTEGER, uuid VARCHAR UNIQUE, wid INTEGER, current_wid INTEGER, origin_x DOUBLE, origin_y DOUBLE, origin_z DOUBLE, x DOUBLE, y DOUBLE, z DOUBLE, yaw FLOAT, pitch FLOAT, data BLOB, removed TINYINT, UNIQUE(kill_rowid))");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "entity_map (" + rowId(prefix, "entity_map") + ", id INTEGER, entity VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "material_map (" + rowId(prefix, "material_map") + ", id INTEGER, material VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "blockdata_map (" + rowId(prefix, "blockdata_map") + ", id INTEGER, data VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "session (" + rowId(prefix, "session") + ", time INTEGER, \"user\" INTEGER, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, action TINYINT)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "sign (" + rowId(prefix, "sign") + ", time INTEGER, \"user\" INTEGER, wid INTEGER, x INTEGER, y INTEGER, z INTEGER, action TINYINT, color INTEGER, color_secondary INTEGER, data TINYINT, waxed TINYINT, face TINYINT, line_1 VARCHAR, line_2 VARCHAR, line_3 VARCHAR, line_4 VARCHAR, line_5 VARCHAR, line_6 VARCHAR, line_7 VARCHAR, line_8 VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "skull (" + rowId(prefix, "skull") + ", time INTEGER, owner VARCHAR, skin VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "user (" + rowId(prefix, "user") + ", time INTEGER, \"user\" VARCHAR, uuid VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "username_log (" + rowId(prefix, "username_log") + ", time INTEGER, uuid VARCHAR, \"user\" VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "version (" + rowId(prefix, "version") + ", time INTEGER, version VARCHAR)");
statement.executeUpdate("CREATE TABLE IF NOT EXISTS " + prefix + "world (" + rowId(prefix, "world") + ", id INTEGER, world VARCHAR)");
if (!purge) {
initialize(prefix, statement);
}
}
}
finally {
if (forceConnection == null) {
connection.close();
}
}
}
private static void initialize(String prefix, Statement statement) throws Exception {
try (ResultSet resultSet = statement.executeQuery("SELECT 1 FROM " + prefix + "database_lock WHERE rowid=1 LIMIT 1")) {
if (!resultSet.next()) {
int timestamp = (int) (System.currentTimeMillis() / 1000L);
statement.executeUpdate("INSERT INTO " + prefix + "database_lock (rowid,status,time) VALUES (1,0," + timestamp + ")");
Process.lastLockUpdate = 0;
}
}
}
private static String rowId(String prefix, String table) {
String type = table.equals("block") ? "BIGINT" : "INTEGER";
return "rowid " + type + " NOT NULL DEFAULT nextval('" + sequence(prefix, table) + "')";
}
private static String sequence(String prefix, String table) {
return prefix + table + "_rowid_seq";
}
}
src/main/java/net/coreprotect/database/DuckDBNativeSupport.java
+42
View File
@@ -0,0 +1,42 @@
package net.coreprotect.database;
import java.io.FileNotFoundException;
import java.sql.Connection;
import java.sql.DriverManager;
public final class DuckDBNativeSupport {
private static final String NATIVE_CLASS = "org.duckdb.DuckDBNative";
private DuckDBNativeSupport() {
throw new IllegalStateException("DuckDBNativeSupport class");
}
public static void verifyAvailable() throws Exception {
Class.forName("org.duckdb.DuckDBDriver");
try (Connection connection = DriverManager.getConnection("jdbc:duckdb:")) {
if (connection == null) {
throw new IllegalStateException("DuckDB JDBC driver did not create an in-memory connection");
}
}
}
public static boolean isNativeUnavailable(Throwable failure) {
for (Throwable cause = failure; cause != null; cause = cause.getCause()) {
if (cause instanceof UnsatisfiedLinkError) {
return true;
}
String message = cause.getMessage();
if (cause instanceof FileNotFoundException && message != null && message.startsWith("DuckDB JNI library not found, path:")) {
return true;
}
if (cause instanceof IllegalStateException && message != null && (message.startsWith("Unsupported OS:") || message.startsWith("Unsupported system architecture:"))) {
return true;
}
if (cause instanceof NoClassDefFoundError && message != null && message.contains(NATIVE_CLASS)) {
return true;
}
}
return false;
}
}
src/main/java/net/coreprotect/database/EntitySpawnUpdateCoordinator.java
+399
View File
@@ -0,0 +1,399 @@
package net.coreprotect.database;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.UUID;
import org.bukkit.Location;
import net.coreprotect.consumer.Queue;
import net.coreprotect.database.rollback.EntitySpawnRollbackHandler;
import net.coreprotect.model.entity.EntityContainerRollbackUpdate;
import net.coreprotect.model.entity.EntitySpawnData;
import net.coreprotect.utility.EntitySpawnTracking;
import net.coreprotect.utility.ErrorReporter;
public final class EntitySpawnUpdateCoordinator {
private final Map<UUID, LocationConfirmation> locationConfirmations = new LinkedHashMap<>();
private final Map<UUID, Long> verificationConfirmations = new LinkedHashMap<>();
private final Set<UUID> missingRows = new HashSet<>();
private final List<EntitySpawnData> lifecycleData = new ArrayList<>();
private final Set<EntitySpawnData> appliedLifecycleData = new HashSet<>();
private final Set<UUID> deferredLifecycleUuids = new HashSet<>();
private final Map<Integer, EntitySpawnData> transitionData = new LinkedHashMap<>();
private final Set<Integer> transitionRows = new HashSet<>();
private final Set<Integer> permanentTransitionRows = new HashSet<>();
private final Map<Integer, EntityContainerRollbackUpdate> combinedTransitionData = new LinkedHashMap<>();
private final Set<Integer> combinedTransitionRows = new HashSet<>();
private final Set<Integer> permanentCombinedTransitionRows = new HashSet<>();
public boolean begin(EntitySpawnData data) {
Objects.requireNonNull(data, "data");
if (isLifecycle(data)) {
lifecycleData.add(data);
if (dependsOnDeferredLifecycle(data)) {
deferLifecycle(data);
return false;
}
}
if (isTransition(data)) {
int trackingRowId = data.getTrackingRowId();
transitionData.put(trackingRowId, data);
transitionRows.remove(trackingRowId);
permanentTransitionRows.remove(trackingRowId);
}
return true;
}
public void applied(EntitySpawnData data) {
if (isLifecycle(data)) {
appliedLifecycleData.add(data);
}
}
public void failed(EntitySpawnData data) {
if (isLifecycle(data) || introducesTrackedUuid(data)) {
deferLifecycle(data);
}
}
public boolean permanentTransitionFailed(EntitySpawnData data, Exception failure) {
if (failure.getSuppressed().length == 0) {
permanentTransitionRows.add(data.getTrackingRowId());
return true;
}
failed(data);
return false;
}
public void beginCombined(EntityContainerRollbackUpdate update) {
EntitySpawnData data = update.getTransition();
int trackingRowId = data.getTrackingRowId();
combinedTransitionData.put(trackingRowId, update);
combinedTransitionRows.remove(trackingRowId);
permanentCombinedTransitionRows.remove(trackingRowId);
}
public void combinedApplied(int trackingRowId) {
transitionRows.remove(trackingRowId);
combinedTransitionRows.add(trackingRowId);
}
public boolean permanentCombinedFailed(EntitySpawnData data, Exception failure) {
transitionRows.remove(data.getTrackingRowId());
if (failure.getSuppressed().length == 0) {
permanentCombinedTransitionRows.add(data.getTrackingRowId());
return true;
}
failed(data);
return false;
}
public void combinedFailed(EntitySpawnData data) {
transitionRows.remove(data.getTrackingRowId());
failed(data);
}
public void transitionApplied(int trackingRowId) {
transitionRows.add(trackingRowId);
}
public void verificationFound(EntitySpawnData data) {
UUID uuid = data.getUuid();
missingRows.remove(uuid);
verificationConfirmations.put(uuid, data.getVerificationEpoch());
}
public void verificationMissing(UUID uuid) {
verificationConfirmations.remove(uuid);
missingRows.add(uuid);
}
public void locationFound(EntitySpawnData data) {
UUID uuid = data.getUuid();
missingRows.remove(uuid);
locationConfirmations.put(uuid, new LocationConfirmation(data));
}
public void locationMissing(UUID uuid) {
locationConfirmations.remove(uuid);
missingRows.add(uuid);
}
public void entityFound(UUID uuid) {
missingRows.remove(uuid);
}
public void entityMissing(UUID uuid) {
missingRows.add(uuid);
}
public void afterCommit(boolean committed) {
List<EntitySpawnData> lifecycleRetries = new ArrayList<>();
List<EntityContainerRollbackUpdate> combinedRetries = new ArrayList<>();
Set<UUID> deferredUuids = new HashSet<>();
for (EntitySpawnData data : lifecycleData) {
if (!committed || !appliedLifecycleData.contains(data)) {
lifecycleRetries.add(data);
deferLifecycle(data);
addDeferredUuid(data, deferredUuids);
}
}
for (Map.Entry<Integer, EntitySpawnData> transition : transitionData.entrySet()) {
EntitySpawnData data = transition.getValue();
if (!permanentTransitionRows.contains(transition.getKey()) && introducesTrackedUuid(data) && (!committed || !transitionRows.contains(transition.getKey()))) {
deferLifecycle(data);
deferredUuids.add(data.getUuid());
}
}
for (Map.Entry<Integer, EntityContainerRollbackUpdate> transition : combinedTransitionData.entrySet()) {
EntitySpawnData data = transition.getValue().getTransition();
if (!permanentCombinedTransitionRows.contains(transition.getKey()) && introducesTrackedUuid(data) && (!committed || !combinedTransitionRows.contains(transition.getKey()))) {
deferLifecycle(data);
deferredUuids.add(data.getUuid());
}
}
try {
if (committed) {
confirmTracking(deferredUuids);
}
}
finally {
List<EntitySpawnData> retries = new ArrayList<>();
List<EntitySpawnData> transitionRetries = new ArrayList<>();
for (Map.Entry<Integer, EntitySpawnData> transition : transitionData.entrySet()) {
int trackingRowId = transition.getKey();
EntitySpawnData data = transition.getValue();
if (permanentTransitionRows.contains(trackingRowId)) {
clearTracking(data);
EntitySpawnRollbackHandler.releaseTrackingRow(trackingRowId);
}
else if (committed && transitionRows.contains(trackingRowId)) {
EntitySpawnRollbackHandler.releaseTrackingRow(trackingRowId);
}
else {
retries.add(data);
transitionRetries.add(data);
}
}
for (Map.Entry<Integer, EntityContainerRollbackUpdate> transition : combinedTransitionData.entrySet()) {
int trackingRowId = transition.getKey();
EntityContainerRollbackUpdate update = transition.getValue();
EntitySpawnData data = update.getTransition();
if (permanentCombinedTransitionRows.contains(trackingRowId)) {
clearTracking(data);
EntitySpawnRollbackHandler.releaseTrackingRow(trackingRowId);
}
else if (committed && combinedTransitionRows.contains(trackingRowId)) {
EntitySpawnRollbackHandler.releaseTrackingRow(trackingRowId);
}
else {
combinedRetries.add(update);
}
}
retries.addAll(lifecycleRetries);
queueRetries(combinedRetries, retries, transitionRetries);
clearPending();
}
}
public void afterRetain() {
clearPending();
deferredLifecycleUuids.clear();
}
public Checkpoint checkpoint() {
return new Checkpoint(this);
}
public void restore(Checkpoint checkpoint) {
Objects.requireNonNull(checkpoint, "checkpoint");
replace(locationConfirmations, checkpoint.locationConfirmations);
replace(verificationConfirmations, checkpoint.verificationConfirmations);
replace(missingRows, checkpoint.missingRows);
replace(lifecycleData, checkpoint.lifecycleData);
replace(appliedLifecycleData, checkpoint.appliedLifecycleData);
replace(deferredLifecycleUuids, checkpoint.deferredLifecycleUuids);
replace(transitionData, checkpoint.transitionData);
replace(transitionRows, checkpoint.transitionRows);
replace(permanentTransitionRows, checkpoint.permanentTransitionRows);
replace(combinedTransitionData, checkpoint.combinedTransitionData);
replace(combinedTransitionRows, checkpoint.combinedTransitionRows);
replace(permanentCombinedTransitionRows, checkpoint.permanentCombinedTransitionRows);
}
public void clearDeferred() {
deferredLifecycleUuids.clear();
}
public static boolean isLifecycle(EntitySpawnData data) {
EntitySpawnData.Operation operation = data.getOperation();
return operation == EntitySpawnData.Operation.VERIFY || operation == EntitySpawnData.Operation.LOCATION || operation == EntitySpawnData.Operation.REMOVED || operation == EntitySpawnData.Operation.REVIVED;
}
private static boolean isTransition(EntitySpawnData data) {
EntitySpawnData.Operation operation = data.getOperation();
return operation == EntitySpawnData.Operation.ROLLBACK || operation == EntitySpawnData.Operation.RESTORE || operation == EntitySpawnData.Operation.KILL_ROLLBACK || operation == EntitySpawnData.Operation.KILL_RESTORE || operation == EntitySpawnData.Operation.COMPOSITE_ROLLBACK || operation == EntitySpawnData.Operation.COMPOSITE_RESTORE || operation == EntitySpawnData.Operation.CLAIM_RELEASE;
}
private static boolean introducesTrackedUuid(EntitySpawnData data) {
return data.getOperation() == EntitySpawnData.Operation.RESTORE || data.getOperation() == EntitySpawnData.Operation.KILL_ROLLBACK;
}
private boolean dependsOnDeferredLifecycle(EntitySpawnData data) {
return deferredLifecycleUuids.contains(data.getUuid()) || (data.getOperation() == EntitySpawnData.Operation.REVIVED && data.getPreviousUuid() != null && deferredLifecycleUuids.contains(data.getPreviousUuid()));
}
private void deferLifecycle(EntitySpawnData data) {
addDeferredUuid(data, deferredLifecycleUuids);
}
private void confirmTracking(Set<UUID> deferredUuids) {
for (LocationConfirmation confirmation : locationConfirmations.values()) {
if (!deferredUuids.contains(confirmation.uuid)) {
runTrackingConfirmation(() -> EntitySpawnTracking.confirmDatabaseLocation(confirmation.uuid, confirmation.location, confirmation.epoch));
}
}
for (Map.Entry<UUID, Long> confirmation : verificationConfirmations.entrySet()) {
if (!deferredUuids.contains(confirmation.getKey())) {
runTrackingConfirmation(() -> EntitySpawnTracking.confirmDatabaseVerification(confirmation.getKey(), confirmation.getValue()));
}
}
for (UUID uuid : missingRows) {
if (!deferredUuids.contains(uuid)) {
runTrackingConfirmation(() -> EntitySpawnTracking.clearTracking(uuid));
}
}
}
private static void queueRetries(List<EntityContainerRollbackUpdate> combinedRetries, List<EntitySpawnData> retries, List<EntitySpawnData> transitionRetries) {
try {
Queue.queueEntityRetriesFirst(combinedRetries, retries);
}
catch (Exception exception) {
for (EntitySpawnData data : transitionRetries) {
cleanupFailedRetry(data, exception);
}
for (EntityContainerRollbackUpdate update : combinedRetries) {
cleanupFailedRetry(update.getTransition(), exception);
}
ErrorReporter.report(exception);
}
}
private static void cleanupFailedRetry(EntitySpawnData data, Exception failure) {
try {
EntitySpawnRollbackHandler.releaseTrackingRow(data.getTrackingRowId());
if (data.getUuid() != null) {
EntitySpawnTracking.clearTracking(data.getUuid());
}
}
catch (Exception cleanupException) {
failure.addSuppressed(cleanupException);
}
}
private static void clearTracking(EntitySpawnData data) {
if (data.getUuid() != null) {
runTrackingConfirmation(() -> EntitySpawnTracking.clearTracking(data.getUuid()));
}
}
private void clearPending() {
locationConfirmations.clear();
verificationConfirmations.clear();
missingRows.clear();
lifecycleData.clear();
appliedLifecycleData.clear();
transitionData.clear();
transitionRows.clear();
permanentTransitionRows.clear();
combinedTransitionData.clear();
combinedTransitionRows.clear();
permanentCombinedTransitionRows.clear();
}
private static void addDeferredUuid(EntitySpawnData data, Set<UUID> uuids) {
if (data.getUuid() != null) {
uuids.add(data.getUuid());
}
if (data.getOperation() == EntitySpawnData.Operation.REVIVED && data.getPreviousUuid() != null) {
uuids.add(data.getPreviousUuid());
}
}
private static void runTrackingConfirmation(Runnable confirmation) {
try {
confirmation.run();
}
catch (Exception exception) {
ErrorReporter.report(exception);
}
}
private static <K, V> void replace(Map<K, V> target, Map<K, V> source) {
target.clear();
target.putAll(source);
}
private static <E> void replace(Set<E> target, Set<E> source) {
target.clear();
target.addAll(source);
}
private static <E> void replace(List<E> target, List<E> source) {
target.clear();
target.addAll(source);
}
public static final class Checkpoint {
private final Map<UUID, LocationConfirmation> locationConfirmations;
private final Map<UUID, Long> verificationConfirmations;
private final Set<UUID> missingRows;
private final List<EntitySpawnData> lifecycleData;
private final Set<EntitySpawnData> appliedLifecycleData;
private final Set<UUID> deferredLifecycleUuids;
private final Map<Integer, EntitySpawnData> transitionData;
private final Set<Integer> transitionRows;
private final Set<Integer> permanentTransitionRows;
private final Map<Integer, EntityContainerRollbackUpdate> combinedTransitionData;
private final Set<Integer> combinedTransitionRows;
private final Set<Integer> permanentCombinedTransitionRows;
private Checkpoint(EntitySpawnUpdateCoordinator coordinator) {
locationConfirmations = new LinkedHashMap<>(coordinator.locationConfirmations);
verificationConfirmations = new LinkedHashMap<>(coordinator.verificationConfirmations);
missingRows = new HashSet<>(coordinator.missingRows);
lifecycleData = new ArrayList<>(coordinator.lifecycleData);
appliedLifecycleData = new HashSet<>(coordinator.appliedLifecycleData);
deferredLifecycleUuids = new HashSet<>(coordinator.deferredLifecycleUuids);
transitionData = new LinkedHashMap<>(coordinator.transitionData);
transitionRows = new HashSet<>(coordinator.transitionRows);
permanentTransitionRows = new HashSet<>(coordinator.permanentTransitionRows);
combinedTransitionData = new LinkedHashMap<>(coordinator.combinedTransitionData);
combinedTransitionRows = new HashSet<>(coordinator.combinedTransitionRows);
permanentCombinedTransitionRows = new HashSet<>(coordinator.permanentCombinedTransitionRows);
}
}
private static final class LocationConfirmation {
private final UUID uuid;
private final Location location;
private final long epoch;
private LocationConfirmation(EntitySpawnData data) {
uuid = data.getUuid();
location = data.getLocation();
epoch = data.getVerificationEpoch();
}
}
}
src/main/java/net/coreprotect/database/Lookup.java
+17 -9
View File
@@ -48,12 +48,14 @@ public class Lookup extends Queue {
public static long countLookupRows(Statement statement, CommandSender user, List<String> checkUuids, List<String> checkUsers, List<Object> restrictList, Map<Object, Boolean> excludeList, List<String> excludeUserList, List<Integer> actionList, EntityActionFilter entityActionFilter, List<String> messageFilters, Set<UUID> loadedEntityUuids, Set<UUID> loadedEntityCandidates, Location location, Integer[] radius, Long[] rowData, long startTime, long endTime, boolean restrictWorld, boolean lookup, Integer entityContainerId, LookupRollbackState rollbackState) {
Long rows = 0L;
boolean paused = false;
try {
while (Consumer.isPaused) {
while (Consumer.isPaused && !Consumer.isPersistenceHalted()) {
Thread.sleep(1);
}
Consumer.isPaused = true;
paused = true;
ResultSet results = LookupRaw.rawLookupResultSet(statement, user, checkUuids, checkUsers, restrictList, excludeList, excludeUserList, actionList, entityActionFilter, messageFilters, loadedEntityUuids, loadedEntityCandidates, location, radius, null, startTime, endTime, -1, -1, restrictWorld, lookup, true, entityContainerId, rollbackState);
while (results.next()) {
@@ -69,8 +71,11 @@ public class Lookup extends Queue {
catch (Exception e) {
ErrorReporter.report(e);
}
Consumer.isPaused = false;
finally {
if (paused && !Consumer.isPersistenceHalted()) {
Consumer.isPaused = false;
}
}
return rows;
}
@@ -86,7 +91,7 @@ public class Lookup extends Queue {
boolean paused = false;
try {
while (Consumer.isPaused) {
while (Consumer.isPaused && !Consumer.isPersistenceHalted()) {
Thread.sleep(1);
}
Consumer.isPaused = true;
@@ -100,7 +105,7 @@ public class Lookup extends Queue {
return 0L;
}
finally {
if (paused) {
if (paused && !Consumer.isPersistenceHalted()) {
Consumer.isPaused = false;
}
}
@@ -118,7 +123,7 @@ public class Lookup extends Queue {
List<LookupSummaryRow> rows = new ArrayList<>();
boolean paused = false;
try {
while (Consumer.isPaused) {
while (Consumer.isPaused && !Consumer.isPersistenceHalted()) {
Thread.sleep(1);
}
Consumer.isPaused = true;
@@ -133,7 +138,7 @@ public class Lookup extends Queue {
ErrorReporter.report(e);
}
finally {
if (paused) {
if (paused && !Consumer.isPersistenceHalted()) {
Consumer.isPaused = false;
}
}
@@ -153,7 +158,7 @@ public class Lookup extends Queue {
long totalRows = 0L;
boolean paused = false;
try {
while (Consumer.isPaused) {
while (Consumer.isPaused && !Consumer.isPersistenceHalted()) {
Thread.sleep(1);
}
Consumer.isPaused = true;
@@ -171,7 +176,7 @@ public class Lookup extends Queue {
ErrorReporter.report(e);
}
finally {
if (paused) {
if (paused && !Consumer.isPersistenceHalted()) {
Consumer.isPaused = false;
}
}
@@ -182,6 +187,9 @@ public class Lookup extends Queue {
try {
DatabaseMetaData metadata = statement.getConnection().getMetaData();
String product = metadata.getDatabaseProductName().toLowerCase(java.util.Locale.ROOT);
if (product.contains("clickhouse") || product.contains("duckdb")) {
return true;
}
int major = metadata.getDatabaseMajorVersion();
int minor = metadata.getDatabaseMinorVersion();
if (product.contains("sqlite")) {
src/main/java/net/coreprotect/database/LookupConverter.java
+1 -4
View File
@@ -29,10 +29,7 @@ public class LookupConverter {
if (i == 2) {
if (map[i] instanceof Integer) {
int userId = (Integer) map[i];
if (ConfigHandler.playerIdCacheReversed.get(userId) == null) {
UserStatement.loadName(statement.getConnection(), userId);
}
String userResult = ConfigHandler.playerIdCacheReversed.get(userId);
String userResult = UserStatement.getName(statement.getConnection(), userId);
results[newId] = userResult;
}
else {
src/main/java/net/coreprotect/database/LookupRaw.java
+97 -67
View File
@@ -34,6 +34,7 @@ import net.coreprotect.model.lookup.LookupRollbackState;
import net.coreprotect.utility.EntitySpawnTracking;
import net.coreprotect.utility.EntityUtils;
import net.coreprotect.utility.ErrorReporter;
import net.coreprotect.utility.DatabaseUtils;
import net.coreprotect.utility.MaterialUtils;
import net.coreprotect.utility.WorldUtils;
@@ -75,14 +76,18 @@ public class LookupRaw extends Queue {
invalidRollbackActions.clear();
}
boolean paused = false;
try {
while (Consumer.isPaused) {
while (Consumer.isPaused && !Consumer.isPersistenceHalted()) {
Thread.sleep(1);
}
Consumer.isPaused = true;
paused = true;
ResultSet results = rawLookupResultSet(statement, user, checkUuids, checkUsers, restrictList, excludeList, excludeUserList, actionList, entityActionFilter, messageFilters, loadedEntityUuids, loadedEntityCandidates, location, radius, rowData, startTime, endTime, limitOffset, limitCount, restrictWorld, lookup, false, entityContainerId, rollbackState);
if (results == null) {
return null;
}
while (results.next()) {
if (actionList.contains(LookupActions.CHAT) || actionList.contains(LookupActions.COMMAND)) {
@@ -216,15 +221,15 @@ public class LookupRaw extends Queue {
if ((lookup && actionList.size() == 0) || actionList.contains(LookupActions.INTERACTION) || actionList.contains(LookupActions.CONTAINER) || actionList.contains(5) || actionList.contains(LookupActions.ITEM)) {
resultData = results.getInt("data");
resultAmount = results.getInt("amount");
resultMeta = results.getBytes("metadata");
resultMeta = DatabaseUtils.getBytes(results, "metadata");
resultTable = results.getInt("tbl");
resultEntitySpawnId = results.getInt("entity_spawn_rowid");
hasTbl = true;
}
else {
resultData = results.getInt("data");
resultMeta = results.getBytes("meta");
resultBlockData = results.getBytes("blockdata");
resultMeta = DatabaseUtils.getBytes(results, "meta");
resultBlockData = DatabaseUtils.getBytes(results, "blockdata");
}
boolean valid = true;
@@ -244,9 +249,13 @@ public class LookupRaw extends Queue {
}
catch (Exception e) {
ErrorReporter.report(e);
return null;
}
finally {
if (paused && !Consumer.isPersistenceHalted()) {
Consumer.isPaused = false;
}
}
Consumer.isPaused = false;
return list;
}
@@ -298,15 +307,23 @@ public class LookupRaw extends Queue {
if (radius != null) {
restrictWorld = true;
}
boolean requiresLocationWorld = restrictWorld || (radius == null && actionList.contains(5) && entityContainerId == null);
int locationWorldId = -1;
if (requiresLocationWorld) {
if (location == null || location.getWorld() == null) {
throw new IllegalArgumentException("Unable to resolve lookup world");
}
locationWorldId = WorldUtils.getWorldId(location.getWorld().getName());
}
boolean inventoryQuery = LookupActions.isInventoryLookup(actionList);
boolean standardActionLookup = !actionList.contains(LookupActions.CONTAINER) && !actionList.contains(5) && !actionList.contains(LookupActions.ITEM) && !actionList.contains(LookupActions.CHAT) && !actionList.contains(LookupActions.COMMAND) && !actionList.contains(LookupActions.SESSION) && !actionList.contains(LookupActions.USERNAME) && !actionList.contains(LookupActions.SIGN);
boolean includeEntityInteractions = !summary && lookup && standardActionLookup && (actionList.isEmpty() || actionList.contains(LookupActions.INTERACTION));
boolean includeEntityContainers = entityContainerId != null || actionList.contains(LookupActions.CONTAINER) || inventoryQuery || (lookup && actionList.isEmpty());
boolean includeEntitySpawnLocations = entityActionFilter.includesAnySpawn(actionList, lookup || Config.getGlobal().ROLLBACK_ENTITIES);
boolean entitySpawnLocation = restrictWorld && location != null && location.getWorld() != null && includeEntitySpawnLocations;
boolean entityContainerLocation = restrictWorld && location != null && location.getWorld() != null && includeEntityContainers;
boolean entityInteractionLocation = restrictWorld && location != null && location.getWorld() != null && includeEntityInteractions;
boolean entitySpawnLocation = restrictWorld && locationWorldId > 0 && includeEntitySpawnLocations;
boolean entityContainerLocation = restrictWorld && locationWorldId > 0 && includeEntityContainers;
boolean entityInteractionLocation = restrictWorld && locationWorldId > 0 && includeEntityInteractions;
boolean entitySpawnRadius = entitySpawnLocation && radius != null;
boolean entityContainerRadius = entityContainerLocation && radius != null;
boolean currentEntityRadius = !lookup && (entitySpawnRadius || entityContainerRadius);
@@ -324,6 +341,7 @@ public class LookupRaw extends Queue {
String users = "";
String uuids = "";
String excludeUsers = "";
String userColumn = ConfigHandler.databaseType.getUserColumn();
String unionLimit = "";
String index = "";
String query = "";
@@ -353,11 +371,7 @@ public class LookupRaw extends Queue {
for (String checkUser : checkUsers) {
if (!checkUser.equals("#container")) {
if (ConfigHandler.playerIdCache.get(checkUser.toLowerCase(Locale.ROOT)) == null) {
UserStatement.loadId(statement.getConnection(), checkUser, null);
}
int userId = ConfigHandler.playerIdCache.get(checkUser.toLowerCase(Locale.ROOT));
int userId = UserStatement.findId(statement.getConnection(), checkUser);
if (checkUserText.length() == 0) {
checkUserText = checkUserText.append(userId);
}
@@ -467,11 +481,10 @@ public class LookupRaw extends Queue {
StringBuilder excludeUserText = new StringBuilder();
for (String excludeTarget : excludeUserList) {
if (ConfigHandler.playerIdCache.get(excludeTarget.toLowerCase(Locale.ROOT)) == null) {
UserStatement.loadId(statement.getConnection(), excludeTarget, null);
int userId = UserStatement.findId(statement.getConnection(), excludeTarget);
if (userId < 0) {
continue;
}
int userId = ConfigHandler.playerIdCache.get(excludeTarget.toLowerCase(Locale.ROOT));
if (excludeUserText.length() == 0) {
excludeUserText = excludeUserText.append(userId);
}
@@ -564,15 +577,15 @@ public class LookupRaw extends Queue {
Integer ymax = radius[4];
Integer zmin = radius[5];
Integer zmax = radius[6];
bounds = "x >= '" + xmin + "' AND x <= '" + xmax + "' AND z >= '" + zmin + "' AND z <= '" + zmax + "'";
bounds = "x >= " + xmin + " AND x <= " + xmax + " AND z >= " + zmin + " AND z <= " + zmax;
if (ymin != null && ymax != null) {
bounds += " AND y >= '" + ymin + "' AND y <= '" + ymax + "'";
bounds += " AND y >= " + ymin + " AND y <= " + ymax;
}
}
if (entitySpawnLocation || entityContainerLocation || entityInteractionLocation) {
int wid = WorldUtils.getWorldId(location.getWorld().getName());
int wid = locationWorldId;
String originalLocation = "(wid=" + wid + (bounds.isEmpty() ? "" : " AND " + bounds) + ")";
String entityBounds = "";
if (radius != null) {
@@ -586,9 +599,9 @@ public class LookupRaw extends Queue {
entityMinZ = Math.floorDiv(radius[5], 16) << 4;
entityMaxZ = ((long) Math.floorDiv(radius[6], 16) << 4) + 16L;
}
entityBounds = "x >= '" + entityMinX + "' AND x < '" + entityMaxX + "' AND z >= '" + entityMinZ + "' AND z < '" + entityMaxZ + "'";
entityBounds = "x >= " + entityMinX + " AND x < " + entityMaxX + " AND z >= " + entityMinZ + " AND z < " + entityMaxZ;
if (!currentEntityRadius && radius[3] != null && radius[4] != null) {
entityBounds += " AND y >= '" + radius[3] + "' AND y < '" + ((long) radius[4] + 1L) + "'";
entityBounds += " AND y >= " + radius[3] + " AND y < " + ((long) radius[4] + 1L);
}
}
String databaseLocation = "(current_wid=" + wid + (entityBounds.isEmpty() ? "" : " AND " + entityBounds) + ")";
@@ -625,8 +638,7 @@ public class LookupRaw extends Queue {
}
else {
if (restrictWorld) {
int wid = WorldUtils.getWorldId(location.getWorld().getName());
queryBlock = queryBlock + " wid=" + wid + " AND";
queryBlock = queryBlock + " wid=" + locationWorldId + " AND";
}
if (!bounds.isEmpty()) {
queryBlock = queryBlock + " " + bounds + " AND";
@@ -634,13 +646,13 @@ public class LookupRaw extends Queue {
}
if (radius == null && actionList.contains(5) && entityContainerId == null) {
int worldId = WorldUtils.getWorldId(location.getWorld().getName());
int worldId = locationWorldId;
int x = (int) Math.floor(location.getX());
int z = (int) Math.floor(location.getZ());
int x2 = (int) Math.ceil(location.getX());
int z2 = (int) Math.ceil(location.getZ());
queryBlock = queryBlock + " wid=" + worldId + " AND (x = '" + x + "' OR x = '" + x2 + "') AND (z = '" + z + "' OR z = '" + z2 + "') AND y = '" + location.getBlockY() + "' AND";
queryBlock = queryBlock + " wid=" + worldId + " AND (x = " + x + " OR x = " + x2 + ") AND (z = " + z + " OR z = " + z2 + ") AND y = " + location.getBlockY() + " AND";
}
String actionPredicate = "";
@@ -665,19 +677,19 @@ public class LookupRaw extends Queue {
}
if (users.length() > 0) {
queryBlock = queryBlock + " user IN(" + users + ") AND";
queryBlock = queryBlock + " " + userColumn + " IN(" + users + ") AND";
}
if (excludeUsers.length() > 0) {
queryBlock = queryBlock + " user NOT IN(" + excludeUsers + ") AND";
queryBlock = queryBlock + " " + userColumn + " NOT IN(" + excludeUsers + ") AND";
}
if (startTime > 0) {
queryBlock = queryBlock + " time > '" + startTime + "' AND";
queryBlock = queryBlock + " time > " + startTime + " AND";
}
if (endTime > 0) {
queryBlock = queryBlock + " time <= '" + endTime + "' AND";
queryBlock = queryBlock + " time <= " + endTime + " AND";
}
String rollbackPredicate = buildRollbackPredicate(rollbackState, actionList.contains(LookupActions.ITEM));
@@ -686,7 +698,7 @@ public class LookupRaw extends Queue {
}
if (actionList.contains(LookupActions.SIGN)) {
queryBlock = queryBlock + " action = '" + SignActions.PLACE + "' AND (LENGTH(line_1) > 0 OR LENGTH(line_2) > 0 OR LENGTH(line_3) > 0 OR LENGTH(line_4) > 0 OR LENGTH(line_5) > 0 OR LENGTH(line_6) > 0 OR LENGTH(line_7) > 0 OR LENGTH(line_8) > 0) AND";
queryBlock = queryBlock + " action = " + SignActions.PLACE + " AND (LENGTH(line_1) > 0 OR LENGTH(line_2) > 0 OR LENGTH(line_3) > 0 OR LENGTH(line_4) > 0 OR LENGTH(line_5) > 0 OR LENGTH(line_6) > 0 OR LENGTH(line_7) > 0 OR LENGTH(line_8) > 0) AND";
}
if (queryBlock.length() > 0) {
@@ -730,20 +742,20 @@ public class LookupRaw extends Queue {
String baseQuery = ((!includeEntity.isEmpty() || !excludeEntity.isEmpty()) ? queryEntity : queryBlock);
String blockSourceQuery = actionList.size() == 1 && actionList.contains(LookupActions.INTERACTION) ? queryBlock : baseQuery;
if (!summary && limitOffset > -1 && limitCount > -1) {
queryLimit = " LIMIT " + limitOffset + ", " + limitCount + "";
queryLimit = " LIMIT " + limitCount + " OFFSET " + limitOffset;
unionLimit = " ORDER BY time DESC, id DESC LIMIT " + (limitOffset + limitCount) + "";
}
String rows = summary ? "rowid as id,time,user,wid,x,y,z,type,meta as metadata,data,-1 as amount,action,rolled_back,0 as entity_spawn_rowid" : "rowid as id,time,user,wid,x,y,z,action,type,data,meta,blockdata,rolled_back";
String rows = summary ? "rowid as id,time," + userColumn + ",wid,x,y,z,type,meta as metadata,data,-1 as amount,action,rolled_back,0 as entity_spawn_rowid" : "rowid as id,time," + userColumn + ",wid,x,y,z,action,type,data,meta,blockdata,rolled_back";
String queryOrder = " ORDER BY rowid DESC";
if (actionList.contains(LookupActions.CONTAINER) || actionList.contains(5)) {
queryTable = "container";
rows = "rowid as id,time,user,wid,x,y,z,action,type,data,rolled_back,amount,metadata,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,action,type,data,rolled_back,amount,metadata,0 as entity_spawn_rowid";
}
else if (actionList.contains(LookupActions.CHAT) || actionList.contains(LookupActions.COMMAND)) {
queryTable = "chat";
rows = "rowid as id,time,user,message";
rows = "rowid as id,time," + userColumn + ",message";
if (PluginChannelHandshakeListener.getInstance().isPluginChannelPlayer(user)) {
rows += ",wid,x,y,z";
}
@@ -754,30 +766,30 @@ public class LookupRaw extends Queue {
}
else if (actionList.contains(LookupActions.SESSION)) {
queryTable = "session";
rows = "rowid as id,time,user,wid,x,y,z,action";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,action";
}
else if (actionList.contains(LookupActions.USERNAME)) {
queryTable = "username_log";
rows = "rowid as id,time,uuid,user";
rows = "rowid as id,time,uuid," + userColumn;
}
else if (actionList.contains(LookupActions.SIGN)) {
queryTable = "sign";
rows = "rowid as id,time,user,wid,x,y,z,face,line_1,line_2,line_3,line_4,line_5,line_6,line_7,line_8";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,face,line_1,line_2,line_3,line_4,line_5,line_6,line_7,line_8";
}
else if (actionList.contains(LookupActions.ITEM)) {
queryTable = "item";
rows = "rowid as id,time,user,wid,x,y,z,type,data as metadata,0 as data,amount,action,rolled_back,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,data as metadata,0 as data,amount,action,rolled_back,0 as entity_spawn_rowid";
}
if (count) {
rows = "COUNT(*) as count";
queryLimit = " LIMIT 0, 5";
queryLimit = " LIMIT 5 OFFSET 0";
queryOrder = "";
unionLimit = "";
}
String unionSelect = "SELECT * FROM (";
if (Config.getGlobal().MYSQL) {
if (ConfigHandler.databaseType.isMySQL()) {
if (queryTable.equals("block")) {
if (!entitySpawnRadius && radius != null && (radius[2] - radius[1]) <= 50 && (radius[6] - radius[5]) <= 50) {
index = "USE INDEX(wid) ";
@@ -795,7 +807,7 @@ public class LookupRaw extends Queue {
unionSelect = "(";
}
else {
else if (ConfigHandler.databaseType.isSQLite()) {
if (queryTable.equals("block")) {
if (includeBlock.length() > 0 || includeEntity.length() > 0) {
index = "INDEXED BY block_type_index ";
@@ -820,7 +832,7 @@ public class LookupRaw extends Queue {
query = unionSelect + "SELECT '0' as tbl," + rows + " FROM " + ConfigHandler.prefix + "chat WHERE" + chatQuery + unionLimit + ") UNION ALL ";
query += unionSelect + "SELECT '1' as tbl," + rows + " FROM " + ConfigHandler.prefix + "command WHERE" + commandQuery + unionLimit + ")";
if (!count) {
queryOrder = " ORDER BY time DESC, id DESC";
queryOrder = " ORDER BY time DESC, tbl DESC, id DESC";
}
}
else if (chatLookup || commandLookup) {
@@ -833,7 +845,7 @@ public class LookupRaw extends Queue {
boolean itemLookup = inventoryQuery;
if ((lookup && actionList.size() == 0) || (itemLookup && !actionList.contains(LookupActions.BLOCK_BREAK))) {
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,meta as metadata,data,-1 as amount,action,rolled_back,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,meta as metadata,data,-1 as amount,action,rolled_back,0 as entity_spawn_rowid";
}
if (inventoryQuery) {
@@ -845,7 +857,7 @@ public class LookupRaw extends Queue {
}
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,meta as metadata,data,1 as amount,action,rolled_back,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,meta as metadata,data,1 as amount,action,rolled_back,0 as entity_spawn_rowid";
}
}
@@ -859,17 +871,17 @@ public class LookupRaw extends Queue {
if (itemLookup) {
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,metadata,data,amount,action,rolled_back,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,metadata,data,amount,action,rolled_back,0 as entity_spawn_rowid";
}
query = query + unionSelect + "SELECT " + "'1' as tbl," + rows + " FROM " + ConfigHandler.prefix + "container WHERE" + queryNonBlock + unionLimit + ") UNION ALL ";
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,metadata,data,amount,action,rolled_back,entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,metadata,data,amount,action,rolled_back,entity_spawn_rowid";
}
query = query + unionSelect + "SELECT '" + InventorySources.ENTITY_CONTAINER + "' as tbl," + rows + " FROM " + ConfigHandler.prefix + "entity_container WHERE" + queryEntityContainer + unionLimit + ") UNION ALL ";
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,data as metadata,0 as data,amount,action,rolled_back,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,data as metadata,0 as data,amount,action,rolled_back,0 as entity_spawn_rowid";
queryOrder = " ORDER BY time DESC, tbl DESC, id DESC";
}
@@ -882,7 +894,7 @@ public class LookupRaw extends Queue {
if (!itemLookup && (actionList.contains(LookupActions.CONTAINER) || actionList.contains(5))) {
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,metadata,data,amount,action,rolled_back,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,metadata,data,amount,action,rolled_back,0 as entity_spawn_rowid";
}
if (entityContainerId == null) {
query = unionSelect + "SELECT '0' as tbl," + rows + " FROM " + ConfigHandler.prefix + "container WHERE" + queryNonBlock + unionLimit + ")";
@@ -892,7 +904,7 @@ public class LookupRaw extends Queue {
query += " UNION ALL ";
}
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,metadata,data,amount,action,rolled_back,entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,metadata,data,amount,action,rolled_back,entity_spawn_rowid";
}
query += unionSelect + "SELECT '" + InventorySources.ENTITY_CONTAINER + "' as tbl," + rows + " FROM " + ConfigHandler.prefix + "entity_container WHERE" + queryEntityContainer + unionLimit + ")";
}
@@ -904,13 +916,13 @@ public class LookupRaw extends Queue {
if (includeEntityInteractions) {
if (query.isEmpty()) {
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,meta as metadata,data,-1 as amount,action,rolled_back,0 as entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,meta as metadata,data,-1 as amount,action,rolled_back,0 as entity_spawn_rowid";
}
query = unionSelect + "SELECT '0' as tbl," + rows + " FROM " + ConfigHandler.prefix + "block " + index + "WHERE" + blockSourceQuery + unionLimit + ")";
}
if (!count) {
rows = "rowid as id,time,user,wid,x,y,z,type,metadata,action as data,-1 as amount," + LookupActions.INTERACTION + " as action,rolled_back,entity_spawn_rowid";
rows = "rowid as id,time," + userColumn + ",wid,x,y,z,type,metadata,action as data,-1 as amount," + LookupActions.INTERACTION + " as action,rolled_back,entity_spawn_rowid";
}
query += " UNION ALL " + unionSelect + "SELECT '" + InventorySources.ENTITY_INTERACTION + "' as tbl," + rows + " FROM " + ConfigHandler.prefix + "entity_interaction WHERE" + queryEntityInteraction + unionLimit + ")";
if (!count) {
@@ -1000,10 +1012,10 @@ public class LookupRaw extends Queue {
private static String entitySpawnTimeQuery(long startTime, long endTime) {
StringBuilder query = new StringBuilder();
if (startTime > 0) {
query.append(" AND time > '").append(startTime).append("'");
query.append(" AND time > ").append(startTime);
}
if (endTime > 0) {
query.append(" AND time <= '").append(endTime).append("'");
query.append(" AND time <= ").append(endTime);
}
return query.toString();
}
@@ -1022,6 +1034,8 @@ public class LookupRaw extends Queue {
}
String alias = table + "FilterRows";
String likeOperator = ConfigHandler.databaseType.isColumnar() ? " ILIKE " : " LIKE ";
String escapeClause = ConfigHandler.databaseType.isClickHouse() ? "" : " ESCAPE '~'";
StringBuilder query = new StringBuilder(baseQuery)
.append(" AND rowid IN (SELECT ").append(alias).append(".rowid FROM ")
.append(ConfigHandler.prefix).append(table).append(" ").append(alias).append(" WHERE (");
@@ -1030,9 +1044,9 @@ public class LookupRaw extends Queue {
query.append(" OR ");
}
String prefixExpression = Config.getGlobal().MYSQL ? alias + ".message" : "substr(" + alias + ".message,1,16)";
query.append("(").append(prefixExpression).append(" LIKE ? ESCAPE '~' AND ")
.append(alias).append(".message LIKE ? ESCAPE '~')");
String prefixExpression = messagePrefix(alias + ".message");
query.append("(").append(prefixExpression).append(likeOperator).append("?").append(escapeClause).append(" AND ")
.append(alias).append(".message").append(likeOperator).append("?").append(escapeClause).append(")");
String filter = messageFilters.get(index) == null ? "" : messageFilters.get(index);
bindings.add(escapeLike(firstCodePoints(filter, 16)) + "%");
@@ -1047,6 +1061,8 @@ public class LookupRaw extends Queue {
}
String alias = "signFilterRows";
String likeOperator = ConfigHandler.databaseType.isColumnar() ? " ILIKE " : " LIKE ";
String escapeClause = ConfigHandler.databaseType.isClickHouse() ? "" : " ESCAPE '~'";
StringBuilder query = new StringBuilder(baseQuery).append(" AND rowid IN (");
boolean union = false;
for (String filter : messageFilters) {
@@ -1058,12 +1074,12 @@ public class LookupRaw extends Queue {
}
String column = "line_" + line;
String prefixExpression = Config.getGlobal().MYSQL ? alias + "." + column : "substr(" + alias + "." + column + ",1,16)";
String prefixExpression = messagePrefix(alias + "." + column);
query.append("SELECT ").append(alias).append(".rowid FROM ")
.append(ConfigHandler.prefix).append("sign ").append(alias)
.append(" WHERE ").append(alias).append(line <= 4 ? ".face = 0" : ".face <> 0")
.append(" AND (").append(prefixExpression).append(" LIKE ? ESCAPE '~' AND ")
.append(alias).append(".").append(column).append(" LIKE ? ESCAPE '~')");
.append(" AND (").append(prefixExpression).append(likeOperator).append("?").append(escapeClause).append(" AND ")
.append(alias).append(".").append(column).append(likeOperator).append("?").append(escapeClause).append(")");
bindings.add(prefix);
bindings.add(message);
union = true;
@@ -1083,11 +1099,25 @@ public class LookupRaw extends Queue {
return value.substring(0, value.offsetByCodePoints(0, maximum));
}
private static String messagePrefix(String column) {
if (ConfigHandler.databaseType.isMySQL()) {
return column;
}
if (ConfigHandler.databaseType.isClickHouse()) {
return "substringUTF8(" + column + ",1,16)";
}
return "substr(" + column + ",1,16)";
}
private static String escapeLike(String value) {
if (ConfigHandler.databaseType.isClickHouse()) {
return value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_");
}
return value.replace("~", "~~").replace("%", "~%").replace("_", "~_");
}
private static String buildSummaryQuery(String sourceQuery, boolean inventoryQuery, boolean countGroups, boolean includeGroupCount, int limitOffset, int limitCount) {
String userColumn = ConfigHandler.databaseType.getUserColumn();
int blockPositiveAction = inventoryQuery ? LookupActions.BLOCK_BREAK : LookupActions.BLOCK_PLACE;
int transactionPositiveAction = inventoryQuery ? ItemTransactionActions.REMOVE : ItemTransactionActions.ADD;
String positiveActions = transactionPositiveAction + "," + ItemTransactionActions.PICKUP + "," + ItemTransactionActions.REMOVE_ENDER + "," + ItemTransactionActions.CREATE + "," + ItemTransactionActions.BUY;
@@ -1095,18 +1125,18 @@ public class LookupRaw extends Queue {
+ "WHEN action IN(" + positiveActions + ") THEN amount ELSE -amount END";
String eligible = "((amount=-1 AND action IN(" + LookupActions.BLOCK_BREAK + "," + LookupActions.BLOCK_PLACE + ")) OR "
+ "(amount<>-1 AND action BETWEEN " + ItemTransactionActions.REMOVE + " AND " + ItemTransactionActions.BUY + "))";
String contributions = "SELECT user,type," + delta + " AS delta FROM (" + sourceQuery + ") summary_source WHERE " + eligible;
String grouped = "SELECT user,type,SUM(CASE WHEN delta<0 THEN -delta ELSE 0 END) AS removed_amount,"
String contributions = "SELECT " + userColumn + ",type," + delta + " AS delta FROM (" + sourceQuery + ") summary_source WHERE " + eligible;
String grouped = "SELECT " + userColumn + ",type,SUM(CASE WHEN delta<0 THEN -delta ELSE 0 END) AS removed_amount,"
+ "SUM(CASE WHEN delta>0 THEN delta ELSE 0 END) AS placed_amount,SUM(delta) AS net_amount "
+ "FROM (" + contributions + ") summary_contributions GROUP BY user,type";
+ "FROM (" + contributions + ") summary_contributions GROUP BY " + userColumn + ",type";
if (countGroups) {
return "SELECT COUNT(*) AS count FROM (" + grouped + ") summary_groups";
}
String totalCount = includeGroupCount ? ",COUNT(*) OVER() AS total_count" : "";
String query = "SELECT user,type,removed_amount,placed_amount,net_amount AS amount" + totalCount + " FROM (" + grouped + ") summary_groups ORDER BY ABS(net_amount) DESC,user ASC,type ASC";
String query = "SELECT " + userColumn + ",type,removed_amount,placed_amount,net_amount AS amount" + totalCount + " FROM (" + grouped + ") summary_groups ORDER BY ABS(net_amount) DESC," + userColumn + " ASC,type ASC";
if (limitOffset > -1 && limitCount > -1) {
query += " LIMIT " + limitOffset + ", " + limitCount;
query += " LIMIT " + limitCount + " OFFSET " + limitOffset;
}
return query;
}
src/main/java/net/coreprotect/database/PurgePolicy.java
+31
View File
@@ -0,0 +1,31 @@
package net.coreprotect.database;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
public final class PurgePolicy {
private static final List<String> PURGEABLE_TABLES = Collections.unmodifiableList(Arrays.asList("sign", "container", "entity_container", "entity_interaction", "item", "skull", "session", "chat", "command", "entity", "block"));
private static final List<String> WORLD_SCOPED_TABLES = Collections.unmodifiableList(Arrays.asList("sign", "container", "entity_container", "entity_interaction", "item", "session", "chat", "command", "block"));
private PurgePolicy() {
throw new IllegalStateException("Policy class");
}
public static List<String> getPurgeableTables() {
return PURGEABLE_TABLES;
}
public static boolean isPurgeable(String table) {
return PURGEABLE_TABLES.contains(table);
}
public static boolean isWorldScoped(String table) {
return WORLD_SCOPED_TABLES.contains(table);
}
public static boolean supportsBlockRestriction(String table) {
return "block".equals(table);
}
}
src/main/java/net/coreprotect/database/RelationalConsumerWriteBatch.java
+828
View File
@@ -0,0 +1,828 @@
package net.coreprotect.database;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Types;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Objects;
import java.util.UUID;
import org.duckdb.DuckDBAppender;
import org.duckdb.DuckDBConnection;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.statement.EntitySpawnStatement;
import net.coreprotect.utility.ErrorReporter;
public final class RelationalConsumerWriteBatch implements ConsumerWriteBatch {
private static final int SIGN_BATCH = 0;
private static final int BLOCK_BATCH = 1;
private static final int CONTAINER_BATCH = 3;
private static final int ENTITY_CONTAINER_BATCH = 4;
private static final int ITEM_BATCH = 5;
private static final int WORLD_BATCH = 6;
private static final int CHAT_BATCH = 7;
private static final int COMMAND_BATCH = 8;
private static final int SESSION_BATCH = 9;
private static final int MATERIAL_BATCH = 11;
private static final int ART_BATCH = 12;
private static final int ENTITY_MAP_BATCH = 13;
private static final int BLOCK_DATA_BATCH = 14;
private static final int ENTITY_KILL_LINK_BATCH = 15;
private static final int INITIAL_DUCKDB_BLOCK_ID_RESERVATION = 256;
private static final int MAXIMUM_DUCKDB_BLOCK_ID_RESERVATION = 65536;
private static final long[] EMPTY_ROW_IDS = new long[0];
private final Connection connection;
private final DatabaseType databaseType;
private final Statement transactionStatement;
private final PreparedStatement[] batchStatements = new PreparedStatement[16];
private final List<PreparedStatement> statements = new ArrayList<>();
private PreparedStatement blockReturningStatement;
private PreparedStatement skullStatement;
private PreparedStatement entityStatement;
private PreparedStatement entitySpawnStatement;
private PreparedStatement entitySpawnBlockLinkStatement;
private PreparedStatement entitySpawnCheckpointStatement;
private PreparedStatement entityInteractionStatement;
private PreparedStatement userByNameStatement;
private PreparedStatement userByNameOrUuidStatement;
private PreparedStatement userInsertStatement;
private PreparedStatement usernameByUuidStatement;
private PreparedStatement usernameUpdateStatement;
private PreparedStatement usernameHistoryStatement;
private PreparedStatement usernameHistoryInsertStatement;
private PreparedStatement databaseLockStatement;
private PreparedStatement duckDBBlockRowIdStatement;
private DuckDBAppender duckDBBlockAppender;
private long[] duckDBBlockRowIds = EMPTY_ROW_IDS;
private int duckDBBlockRowIdIndex;
private int duckDBBlockIdReservationSize = INITIAL_DUCKDB_BLOCK_ID_RESERVATION;
private EntitySpawnStatement.Updates entitySpawnUpdates;
public RelationalConsumerWriteBatch(Connection connection, DatabaseType databaseType) throws SQLException {
this.connection = Objects.requireNonNull(connection, "connection");
this.databaseType = Objects.requireNonNull(databaseType, "databaseType");
transactionStatement = connection.createStatement();
}
@Override
public void begin() throws Exception {
Database.beginTransaction(transactionStatement, databaseType);
}
@Override
public boolean commit() throws Exception {
try {
finishDuckDBBlockAppender();
boolean acknowledgedRollback = Database.isRollbackOnlyTransactionAcknowledged();
if (!Database.isTransactionRollbackOnly()) {
for (PreparedStatement statement : batchStatements) {
if (statement != null) {
statement.executeBatch();
}
}
}
boolean committed = Database.commitTransactionChecked(transactionStatement, databaseType);
if (!committed) {
boolean rolledBack = Database.rollbackTransaction(transactionStatement, databaseType);
return acknowledgedRollback && rolledBack;
}
return true;
}
catch (Exception exception) {
Database.rollbackTransaction(transactionStatement, databaseType);
ErrorReporter.report(exception);
return false;
}
}
@Override
public void rollback() {
try {
finishDuckDBBlockAppender();
}
catch (Exception exception) {
ErrorReporter.report(exception);
}
Database.rollbackTransaction(transactionStatement, databaseType);
}
@Override
public void executeAtomically(String name, Database.SavepointOperation operation) throws Exception {
Database.executeSavepoint(transactionStatement, name, operation);
}
@Override
public int resolveUserId(String user, String uuid) throws Exception {
String cacheKey = user.toLowerCase(Locale.ROOT);
Integer cachedId = ConfigHandler.playerIdCache.get(cacheKey);
if (cachedId != null) {
return cachedId;
}
PreparedStatement statement = uuid == null ? userByNameStatement() : userByNameOrUuidStatement();
statement.setString(1, user);
if (uuid != null) {
statement.setString(2, uuid);
}
int userId = -1;
String storedUuid = uuid;
try (ResultSet resultSet = statement.executeQuery()) {
if (resultSet.next()) {
userId = resultSet.getInt(1);
storedUuid = resultSet.getString(2);
}
}
if (userId == -1) {
userId = insertUser(user);
}
ConfigHandler.playerIdCache.put(cacheKey, userId);
ConfigHandler.playerIdCacheReversed.put(userId, user);
if (storedUuid != null) {
ConfigHandler.uuidCache.put(cacheKey, storedUuid);
ConfigHandler.uuidCacheReversed.put(storedUuid, user);
}
return userId;
}
@Override
public void recordUsername(String user, String uuid, int retainHistory, int time) throws Exception {
if (ConfigHandler.isBlacklisted(user)) {
return;
}
PreparedStatement lookup = usernameByUuidStatement();
lookup.setString(1, uuid);
int userId = -1;
String storedUser = null;
try (ResultSet resultSet = lookup.executeQuery()) {
if (resultSet.next()) {
userId = resultSet.getInt(1);
storedUser = resultSet.getString(2);
}
}
boolean update = storedUser == null || !user.equalsIgnoreCase(storedUser);
if (storedUser == null) {
Integer cachedId = ConfigHandler.playerIdCache.get(user.toLowerCase(Locale.ROOT));
userId = cachedId == null ? resolveUserId(user, uuid) : cachedId;
}
if (update) {
PreparedStatement statement = usernameUpdateStatement();
statement.setString(1, user);
statement.setString(2, uuid);
statement.setInt(3, userId);
statement.executeUpdate();
}
else {
PreparedStatement statement = usernameHistoryStatement();
statement.setString(1, uuid);
statement.setString(2, user);
try (ResultSet resultSet = statement.executeQuery()) {
update = !resultSet.next();
}
}
if (update && retainHistory == 1) {
PreparedStatement statement = usernameHistoryInsertStatement();
statement.setInt(1, time);
statement.setString(2, uuid);
statement.setString(3, user);
statement.executeUpdate();
}
String cacheKey = user.toLowerCase(Locale.ROOT);
ConfigHandler.playerIdCache.put(cacheKey, userId);
ConfigHandler.playerIdCacheReversed.put(userId, user);
ConfigHandler.uuidCache.put(cacheKey, uuid);
ConfigHandler.uuidCacheReversed.put(uuid, user);
}
@Override
public void updateDatabaseLock(int status, int time) throws Exception {
PreparedStatement statement = databaseLockStatement();
statement.setInt(1, status);
statement.setInt(2, time);
statement.executeUpdate();
}
@Override
public void addReference(ReferenceKind kind, int batchCount, int id, String value) throws Exception {
PreparedStatement statement;
switch (kind) {
case ART:
statement = batchStatement(Database.ART, ART_BATCH);
break;
case BLOCK_DATA:
statement = batchStatement(Database.BLOCKDATA, BLOCK_DATA_BATCH);
break;
case ENTITY:
statement = batchStatement(Database.ENTITY_MAP, ENTITY_MAP_BATCH);
break;
case MATERIAL:
statement = batchStatement(Database.MATERIAL, MATERIAL_BATCH);
break;
case WORLD:
statement = batchStatement(Database.WORLD, WORLD_BATCH);
break;
default:
throw new IllegalArgumentException("Unsupported reference kind " + kind);
}
statement.setInt(1, id);
statement.setString(2, value);
addBatch(statement, batchCount);
}
@Override
public void addBlock(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws Exception {
if (databaseType.isDuckDB()) {
appendDuckDBBlock(time, userId, worldId, x, y, z, type, data, meta, blockData, action, rolledBack);
return;
}
PreparedStatement statement = batchStatement(Database.BLOCK, BLOCK_BATCH);
setBlock(statement, time, userId, worldId, x, y, z, type, data, meta, blockData, action, rolledBack);
addBatch(statement, batchCount);
}
@Override
public long addBlockReturningId(int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws Exception {
if (databaseType.isDuckDB()) {
flushDuckDBBlockAppender();
resetDuckDBBlockRowIds();
}
if (blockReturningStatement == null) {
blockReturningStatement = own(required(Database.prepareStatement(connection, Database.BLOCK, true), "block insert"));
}
setBlock(blockReturningStatement, time, userId, worldId, x, y, z, type, data, meta, blockData, action, rolledBack);
return executeReturningId(blockReturningStatement, "block insert");
}
@Override
public int addSkull(int time, String owner, String skin) throws Exception {
if (skullStatement == null) {
skullStatement = own(required(Database.prepareStatement(connection, Database.SKULL, true), "skull insert"));
}
skullStatement.setInt(1, time);
skullStatement.setString(2, owner);
skullStatement.setString(3, skin);
return Math.toIntExact(executeReturningId(skullStatement, "skull insert"));
}
@Override
public void addContainer(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws Exception {
PreparedStatement statement = batchStatement(Database.CONTAINER, CONTAINER_BATCH);
statement.setInt(1, time);
statement.setInt(2, userId);
statement.setInt(3, worldId);
statement.setInt(4, x);
statement.setInt(5, y);
statement.setInt(6, z);
statement.setInt(7, type);
statement.setInt(8, data);
statement.setInt(9, amount);
statement.setObject(10, metadata);
statement.setInt(11, action);
statement.setInt(12, rolledBack);
addBatch(statement, batchCount);
}
@Override
public void addEntityContainer(int batchCount, int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws Exception {
PreparedStatement statement = batchStatement(Database.ENTITY_CONTAINER, ENTITY_CONTAINER_BATCH);
statement.setInt(1, time);
statement.setInt(2, userId);
statement.setInt(3, entitySpawnRowId);
statement.setInt(4, worldId);
statement.setInt(5, x);
statement.setInt(6, y);
statement.setInt(7, z);
statement.setInt(8, type);
statement.setInt(9, data);
statement.setInt(10, amount);
statement.setObject(11, metadata);
statement.setInt(12, action);
statement.setInt(13, rolledBack);
addBatch(statement, batchCount);
}
@Override
public void addItem(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, byte[] data, int amount, int action, int rolledBack) throws Exception {
PreparedStatement statement = batchStatement(Database.ITEM, ITEM_BATCH);
statement.setInt(1, time);
statement.setInt(2, userId);
statement.setInt(3, worldId);
statement.setInt(4, x);
statement.setInt(5, y);
statement.setInt(6, z);
statement.setInt(7, type);
statement.setObject(8, data);
statement.setInt(9, amount);
statement.setInt(10, action);
statement.setInt(11, rolledBack);
addBatch(statement, batchCount);
}
@Override
public void addChat(int batchCount, long time, int userId, int worldId, int x, int y, int z, String message) throws Exception {
PreparedStatement statement = batchStatement(Database.CHAT, CHAT_BATCH);
setMessage(statement, time, userId, worldId, x, y, z, message);
addBatch(statement, batchCount);
}
@Override
public void addCommand(int batchCount, long time, int userId, int worldId, int x, int y, int z, String message) throws Exception {
PreparedStatement statement = batchStatement(Database.COMMAND, COMMAND_BATCH);
setMessage(statement, time, userId, worldId, x, y, z, message);
addBatch(statement, batchCount);
}
@Override
public void addSession(int batchCount, int time, int userId, int worldId, int x, int y, int z, int action) throws Exception {
PreparedStatement statement = batchStatement(Database.SESSION, SESSION_BATCH);
statement.setInt(1, time);
statement.setInt(2, userId);
statement.setInt(3, worldId);
statement.setInt(4, x);
statement.setInt(5, y);
statement.setInt(6, z);
statement.setInt(7, action);
addBatch(statement, batchCount);
}
@Override
public void addSign(int batchCount, int time, int userId, int worldId, int x, int y, int z, int action, int color, int colorSecondary, int data, int waxed, int face, String[] lines) throws Exception {
if (lines.length != 8) {
throw new IllegalArgumentException("Sign data must contain eight lines");
}
PreparedStatement statement = batchStatement(Database.SIGN, SIGN_BATCH);
statement.setInt(1, time);
statement.setInt(2, userId);
statement.setInt(3, worldId);
statement.setInt(4, x);
statement.setInt(5, y);
statement.setInt(6, z);
statement.setInt(7, action);
statement.setInt(8, color);
statement.setInt(9, colorSecondary);
statement.setInt(10, data);
statement.setInt(11, waxed);
statement.setInt(12, face);
for (int index = 0; index < lines.length; index++) {
statement.setString(13 + index, lines[index]);
}
addBatch(statement, batchCount);
}
@Override
public int addEntity(int time, byte[] data) throws Exception {
if (entityStatement == null) {
entityStatement = own(required(Database.prepareStatement(connection, Database.ENTITY, true), "entity insert"));
}
entityStatement.setInt(1, time);
entityStatement.setObject(2, data);
return Math.toIntExact(executeReturningId(entityStatement, "entity insert"));
}
@Override
public int addEntitySpawn(int time, Long blockRowId, Integer killRowId, UUID uuid, int originWorldId, int currentWorldId, double originX, double originY, double originZ, double currentX, double currentY, double currentZ, float yaw, float pitch, byte[] data, int removed) throws Exception {
PreparedStatement statement = entitySpawnStatement();
statement.setInt(1, time);
setNullableLong(statement, 2, blockRowId);
setNullableInt(statement, 3, killRowId);
statement.setString(4, uuid.toString());
statement.setInt(5, originWorldId);
statement.setInt(6, currentWorldId);
statement.setDouble(7, originX);
statement.setDouble(8, originY);
statement.setDouble(9, originZ);
statement.setDouble(10, currentX);
statement.setDouble(11, currentY);
statement.setDouble(12, currentZ);
statement.setFloat(13, yaw);
statement.setFloat(14, pitch);
if (data == null) {
statement.setNull(15, Types.BLOB);
}
else {
statement.setBytes(15, data);
}
statement.setInt(16, removed);
return Math.toIntExact(executeReturningId(statement, "entity spawn insert"));
}
@Override
public void linkEntitySpawnBlock(int trackingRowId, long blockRowId) throws Exception {
if (entitySpawnBlockLinkStatement == null) {
entitySpawnBlockLinkStatement = own(connection.prepareStatement("UPDATE " + ConfigHandler.prefix + "entity_spawn SET block_rowid=? WHERE rowid=? AND block_rowid IS NULL"));
}
entitySpawnBlockLinkStatement.setLong(1, blockRowId);
entitySpawnBlockLinkStatement.setInt(2, trackingRowId);
if (entitySpawnBlockLinkStatement.executeUpdate() != 1) {
throw new SQLException("Entity spawn tracking row did not link to its block row");
}
}
@Override
public void linkEntitySpawnKill(UUID uuid, int killRowId) throws Exception {
PreparedStatement statement = batchStatements[ENTITY_KILL_LINK_BATCH];
if (statement == null) {
statement = own(connection.prepareStatement("UPDATE " + ConfigHandler.prefix + "entity_spawn SET kill_rowid=? WHERE uuid=?"));
batchStatements[ENTITY_KILL_LINK_BATCH] = statement;
}
statement.setInt(1, killRowId);
statement.setString(2, uuid.toString());
statement.addBatch();
}
@Override
public void checkpointEntitySpawn(int trackingRowId, int worldId, double x, double y, double z, float yaw, float pitch) throws Exception {
if (entitySpawnCheckpointStatement == null) {
entitySpawnCheckpointStatement = own(connection.prepareStatement("UPDATE " + ConfigHandler.prefix + "entity_spawn SET current_wid=?,x=?,y=?,z=?,yaw=?,pitch=? WHERE rowid=? AND removed=0"));
}
entitySpawnCheckpointStatement.setInt(1, worldId);
entitySpawnCheckpointStatement.setDouble(2, x);
entitySpawnCheckpointStatement.setDouble(3, y);
entitySpawnCheckpointStatement.setDouble(4, z);
entitySpawnCheckpointStatement.setFloat(5, yaw);
entitySpawnCheckpointStatement.setFloat(6, pitch);
entitySpawnCheckpointStatement.setInt(7, trackingRowId);
if (entitySpawnCheckpointStatement.executeUpdate() != 1) {
throw new SQLException("Entity interaction could not checkpoint its tracking row");
}
}
@Override
public void addEntityInteraction(int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int action, byte[] metadata, int rolledBack) throws Exception {
if (entityInteractionStatement == null) {
entityInteractionStatement = own(required(Database.prepareStatement(connection, Database.ENTITY_INTERACTION, false), "entity interaction insert"));
}
entityInteractionStatement.setInt(1, time);
entityInteractionStatement.setInt(2, userId);
entityInteractionStatement.setInt(3, entitySpawnRowId);
entityInteractionStatement.setInt(4, worldId);
entityInteractionStatement.setInt(5, x);
entityInteractionStatement.setInt(6, y);
entityInteractionStatement.setInt(7, z);
entityInteractionStatement.setInt(8, type);
entityInteractionStatement.setInt(9, action);
if (metadata == null) {
entityInteractionStatement.setNull(10, Types.BLOB);
}
else {
entityInteractionStatement.setBytes(10, metadata);
}
entityInteractionStatement.setInt(11, rolledBack);
if (entityInteractionStatement.executeUpdate() != 1) {
throw new SQLException("Entity interaction insert did not insert one row");
}
}
@Override
public void updateRolledBack(int target, int rolledBack, List<Long> rowIds) throws Exception {
Database.performRolledBackUpdateChecked(transactionStatement, rolledBack, rowIds, target);
}
@Override
public ConsumerEntitySpawnUpdates entitySpawnUpdates() throws Exception {
if (entitySpawnUpdates == null) {
entitySpawnUpdates = new EntitySpawnStatement.Updates(connection);
}
return entitySpawnUpdates;
}
@Override
public void close() throws Exception {
Exception failure = null;
try {
finishDuckDBBlockAppender();
}
catch (Exception exception) {
failure = exception;
}
if (entitySpawnUpdates != null) {
try {
entitySpawnUpdates.close();
}
catch (Exception exception) {
failure = addFailure(failure, exception);
}
}
for (PreparedStatement statement : statements) {
try {
statement.close();
}
catch (Exception exception) {
failure = addFailure(failure, exception);
}
}
try {
transactionStatement.close();
}
catch (Exception exception) {
failure = addFailure(failure, exception);
}
if (failure != null) {
throw failure;
}
}
private PreparedStatement batchStatement(int type, int index) throws SQLException {
PreparedStatement statement = batchStatements[index];
if (statement == null) {
statement = own(required(Database.prepareStatement(connection, type, false), "batched insert"));
batchStatements[index] = statement;
}
return statement;
}
private PreparedStatement entitySpawnStatement() throws SQLException {
if (entitySpawnStatement == null) {
String sql = "INSERT INTO " + ConfigHandler.prefix + "entity_spawn (time,block_rowid,kill_rowid,uuid,wid,current_wid,origin_x,origin_y,origin_z,x,y,z,yaw,pitch,data,removed) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
entitySpawnStatement = own(prepare(sql, true));
}
return entitySpawnStatement;
}
private PreparedStatement userByNameStatement() throws SQLException {
if (userByNameStatement == null) {
String collate = databaseType.isMySQL() ? "" : " COLLATE NOCASE";
userByNameStatement = own(connection.prepareStatement("SELECT rowid,uuid FROM " + ConfigHandler.prefix + "user WHERE " + databaseType.getUserColumn() + "=?" + collate + " ORDER BY rowid ASC LIMIT 1 OFFSET 0"));
}
return userByNameStatement;
}
private PreparedStatement userByNameOrUuidStatement() throws SQLException {
if (userByNameOrUuidStatement == null) {
String collate = databaseType.isMySQL() ? "" : " COLLATE NOCASE";
userByNameOrUuidStatement = own(connection.prepareStatement("SELECT rowid,uuid FROM " + ConfigHandler.prefix + "user WHERE " + databaseType.getUserColumn() + "=?" + collate + " OR uuid=? ORDER BY rowid ASC LIMIT 1 OFFSET 0"));
}
return userByNameOrUuidStatement;
}
private PreparedStatement usernameByUuidStatement() throws SQLException {
if (usernameByUuidStatement == null) {
usernameByUuidStatement = own(connection.prepareStatement("SELECT rowid," + databaseType.getUserColumn() + " FROM " + ConfigHandler.prefix + "user WHERE uuid=? LIMIT 1 OFFSET 0"));
}
return usernameByUuidStatement;
}
private PreparedStatement usernameUpdateStatement() throws SQLException {
if (usernameUpdateStatement == null) {
usernameUpdateStatement = own(connection.prepareStatement("UPDATE " + ConfigHandler.prefix + "user SET " + databaseType.getUserColumn() + "=?,uuid=? WHERE rowid=?"));
}
return usernameUpdateStatement;
}
private PreparedStatement usernameHistoryStatement() throws SQLException {
if (usernameHistoryStatement == null) {
usernameHistoryStatement = own(connection.prepareStatement("SELECT rowid FROM " + ConfigHandler.prefix + "username_log WHERE uuid=? AND " + databaseType.getUserColumn() + "=? LIMIT 1 OFFSET 0"));
}
return usernameHistoryStatement;
}
private PreparedStatement usernameHistoryInsertStatement() throws SQLException {
if (usernameHistoryInsertStatement == null) {
usernameHistoryInsertStatement = own(connection.prepareStatement("INSERT INTO " + ConfigHandler.prefix + "username_log (time,uuid," + databaseType.getUserColumn() + ") VALUES (?,?,?)"));
}
return usernameHistoryInsertStatement;
}
private PreparedStatement databaseLockStatement() throws SQLException {
if (databaseLockStatement == null) {
databaseLockStatement = own(connection.prepareStatement("UPDATE " + ConfigHandler.prefix + "database_lock SET status=?,time=? WHERE rowid=1"));
}
return databaseLockStatement;
}
private void appendDuckDBBlock(int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws SQLException {
long rowId = nextDuckDBBlockRowId();
if (duckDBBlockAppender == null) {
DuckDBConnection duckDBConnection = connection.unwrap(DuckDBConnection.class);
duckDBBlockAppender = duckDBConnection.createAppender(DuckDBConnection.DEFAULT_SCHEMA, ConfigHandler.prefix + "block");
}
duckDBBlockAppender.beginRow()
.append(rowId)
.append(time)
.append(userId)
.append(worldId)
.append(x)
.append(y)
.append(z)
.append(type)
.append(data);
appendNullable(duckDBBlockAppender, meta);
appendNullable(duckDBBlockAppender, blockData);
duckDBBlockAppender.append((byte) action)
.append((byte) rolledBack)
.endRow();
}
private long nextDuckDBBlockRowId() throws SQLException {
if (duckDBBlockRowIdIndex >= duckDBBlockRowIds.length) {
reserveDuckDBBlockRowIds();
}
return duckDBBlockRowIds[duckDBBlockRowIdIndex++];
}
private void reserveDuckDBBlockRowIds() throws SQLException {
flushDuckDBBlockAppender();
if (duckDBBlockRowIdStatement == null) {
String sequence = ConfigHandler.prefix + "block_rowid_seq";
duckDBBlockRowIdStatement = own(connection.prepareStatement("SELECT nextval('" + sequence + "') FROM range(?)"));
}
int reservationSize = duckDBBlockIdReservationSize;
long[] rowIds = new long[reservationSize];
duckDBBlockRowIdStatement.setInt(1, reservationSize);
try (ResultSet resultSet = duckDBBlockRowIdStatement.executeQuery()) {
for (int index = 0; index < reservationSize; index++) {
if (!resultSet.next()) {
throw new SQLException("DuckDB block row id reservation returned too few values");
}
rowIds[index] = resultSet.getLong(1);
}
if (resultSet.next()) {
throw new SQLException("DuckDB block row id reservation returned too many values");
}
}
duckDBBlockRowIds = rowIds;
duckDBBlockRowIdIndex = 0;
duckDBBlockIdReservationSize = Math.min(MAXIMUM_DUCKDB_BLOCK_ID_RESERVATION, reservationSize * 2);
}
private void flushDuckDBBlockAppender() throws SQLException {
if (duckDBBlockAppender != null) {
duckDBBlockAppender.flush();
}
}
private void finishDuckDBBlockAppender() throws SQLException {
DuckDBAppender appender = duckDBBlockAppender;
duckDBBlockAppender = null;
resetDuckDBBlockRowIds();
if (appender == null) {
return;
}
SQLException failure = null;
try {
appender.flush();
}
catch (SQLException exception) {
failure = exception;
}
try {
appender.close();
}
catch (SQLException exception) {
if (failure == null) {
failure = exception;
}
else {
failure.addSuppressed(exception);
}
}
if (failure != null) {
throw failure;
}
}
private void resetDuckDBBlockRowIds() {
duckDBBlockRowIds = EMPTY_ROW_IDS;
duckDBBlockRowIdIndex = 0;
duckDBBlockIdReservationSize = INITIAL_DUCKDB_BLOCK_ID_RESERVATION;
}
private static void appendNullable(DuckDBAppender appender, byte[] value) throws SQLException {
if (value == null) {
appender.appendNull();
}
else {
appender.append(value);
}
}
private int insertUser(String user) throws Exception {
if (userInsertStatement == null) {
String sql = "INSERT INTO " + ConfigHandler.prefix + "user (time," + databaseType.getUserColumn() + ") VALUES (?,?)";
userInsertStatement = own(prepare(sql, true));
}
userInsertStatement.setInt(1, (int) (System.currentTimeMillis() / 1000L));
userInsertStatement.setString(2, user);
return Math.toIntExact(executeReturningId(userInsertStatement, "user insert"));
}
private PreparedStatement prepare(String sql, boolean keys) throws SQLException {
if (!keys) {
return connection.prepareStatement(sql);
}
if (Database.hasReturningKeys()) {
return connection.prepareStatement(sql + " RETURNING rowid");
}
return connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS);
}
private PreparedStatement own(PreparedStatement statement) {
statements.add(statement);
return statement;
}
private static PreparedStatement required(PreparedStatement statement, String operation) throws SQLException {
if (statement == null) {
throw new SQLException("Unable to prepare " + operation);
}
return statement;
}
private static void addBatch(PreparedStatement statement, int batchCount) throws SQLException {
statement.addBatch();
if (batchCount > 0 && batchCount % 1000 == 0) {
statement.executeBatch();
}
}
private static long executeReturningId(PreparedStatement statement, String operation) throws Exception {
if (Database.hasReturningKeys()) {
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException(operation + " did not return a row id");
}
return resultSet.getLong(1);
}
}
int updated = statement.executeUpdate();
if (updated != 1) {
throw new SQLException("Expected one row for " + operation + ", updated " + updated);
}
try (ResultSet resultSet = statement.getGeneratedKeys()) {
if (!resultSet.next()) {
throw new SQLException(operation + " did not generate a row id");
}
return resultSet.getLong(1);
}
}
private static void setBlock(PreparedStatement statement, int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws SQLException {
statement.setInt(1, time);
statement.setInt(2, userId);
statement.setInt(3, worldId);
statement.setInt(4, x);
statement.setInt(5, y);
statement.setInt(6, z);
statement.setInt(7, type);
statement.setInt(8, data);
statement.setObject(9, meta);
statement.setObject(10, blockData);
statement.setInt(11, action);
statement.setInt(12, rolledBack);
}
private static void setMessage(PreparedStatement statement, long time, int userId, int worldId, int x, int y, int z, String message) throws SQLException {
statement.setLong(1, time);
statement.setInt(2, userId);
statement.setInt(3, worldId);
statement.setInt(4, x);
statement.setInt(5, y);
statement.setInt(6, z);
statement.setString(7, message);
}
private static void setNullableLong(PreparedStatement statement, int index, Long value) throws SQLException {
if (value == null) {
statement.setNull(index, Types.BIGINT);
}
else {
statement.setLong(index, value);
}
}
private static void setNullableInt(PreparedStatement statement, int index, Integer value) throws SQLException {
if (value == null) {
statement.setNull(index, Types.INTEGER);
}
else {
statement.setInt(index, value);
}
}
private static Exception addFailure(Exception failure, Exception exception) {
if (failure == null) {
return exception;
}
failure.addSuppressed(exception);
return failure;
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseBatchIdentity.java
+70
View File
@@ -0,0 +1,70 @@
package net.coreprotect.database.clickhouse;
import java.util.Objects;
import java.util.UUID;
public final class ClickHouseBatchIdentity {
private static final String TOKEN_VERSION = "coreprotect-v1";
private final UUID datasetId;
private final UUID producerId;
private final long producerSequence;
private final UUID batchId;
public ClickHouseBatchIdentity(UUID datasetId, UUID producerId, long producerSequence, UUID batchId) {
this.datasetId = Objects.requireNonNull(datasetId, "datasetId");
this.producerId = Objects.requireNonNull(producerId, "producerId");
if (producerSequence < 1) {
throw new IllegalArgumentException("Producer sequence must be positive");
}
this.producerSequence = producerSequence;
this.batchId = Objects.requireNonNull(batchId, "batchId");
}
public static ClickHouseBatchIdentity create(UUID datasetId, UUID producerId, long producerSequence) {
return new ClickHouseBatchIdentity(datasetId, producerId, producerSequence, UUID.randomUUID());
}
public UUID getDatasetId() {
return datasetId;
}
public UUID getProducerId() {
return producerId;
}
public long getProducerSequence() {
return producerSequence;
}
public UUID getBatchId() {
return batchId;
}
public String getDeduplicationToken() {
return TOKEN_VERSION + ":" + datasetId + ":" + producerId + ":" + producerSequence + ":" + batchId;
}
@Override
public boolean equals(Object object) {
if (this == object) {
return true;
}
if (!(object instanceof ClickHouseBatchIdentity)) {
return false;
}
ClickHouseBatchIdentity identity = (ClickHouseBatchIdentity) object;
return producerSequence == identity.producerSequence && datasetId.equals(identity.datasetId) && producerId.equals(identity.producerId) && batchId.equals(identity.batchId);
}
@Override
public int hashCode() {
return Objects.hash(datasetId, producerId, producerSequence, batchId);
}
@Override
public String toString() {
return getDeduplicationToken();
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseBatchPublisher.java
+198
View File
@@ -0,0 +1,198 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Objects;
import net.coreprotect.config.ConfigHandler;
final class ClickHouseBatchPublisher {
private static final long MAX_INSERT_ROWS = 1_000_000;
private static final int MAX_PUBLICATION_ATTEMPTS = 3;
private static final int MAX_RECONCILIATION_ATTEMPTS = 3;
private static final long RETRY_DELAY_MILLIS = 100L;
private static final long MAX_RETRY_DELAY_MILLIS = 30_000L;
private final ClickHouseJdbc jdbc;
private final ClickHouseNativeClient nativeClient;
private final ClickHouseWriterRegistration writerRegistration;
private final String eventTable;
ClickHouseBatchPublisher(ClickHouseJdbc jdbc, ClickHouseNativeClient nativeClient, ClickHouseWriterRegistration writerRegistration, String database, String prefix) {
this.jdbc = Objects.requireNonNull(jdbc, "jdbc");
this.nativeClient = Objects.requireNonNull(nativeClient, "nativeClient");
this.writerRegistration = Objects.requireNonNull(writerRegistration, "writerRegistration");
String validatedPrefix = prefix == null || prefix.isEmpty() ? "" : ClickHouseIdentifiers.requireIdentifier(prefix, "ClickHouse table prefix");
eventTable = ClickHouseIdentifiers.qualified(database, validatedPrefix + "event_data");
}
ClickHouseBatchReceipt publish(ClickHouseWriteBatch batch) throws SQLException {
Objects.requireNonNull(batch, "batch");
ClickHouseBatchReceipt receipt = batch.seal();
if (batch.isPublished()) {
return receipt;
}
if (receipt.getRowCount() == 0) {
batch.markPublished();
return receipt;
}
if (receipt.getRowCount() > MAX_INSERT_ROWS) {
throw new SQLException("ClickHouse batches cannot exceed " + MAX_INSERT_ROWS + " physical rows");
}
SQLException publicationFailure = null;
int attempt = 1;
while (true) {
try {
writerRegistration.verifyOwned();
nativeClient.insert(eventTable, ClickHouseSchema.EVENT_COLUMNS, batch.openRows(), batch.getIdentity(), "records");
batch.markPublished();
return receipt;
}
catch (ClickHouseWriterRegistration.OwnershipException exception) {
throw exception;
}
catch (SQLException exception) {
publicationFailure = exception;
}
RawStatus status;
try {
status = readStatus(batch.getIdentity());
}
catch (SQLException reconciliationFailure) {
reconciliationFailure.addSuppressed(publicationFailure);
throw reconciliationFailure;
}
if (status.matches(receipt)) {
batch.markPublished();
return receipt;
}
if (!status.isEmpty() && !status.isRetryablePartial(receipt)) {
SQLException conflict = conflict(receipt);
conflict.addSuppressed(publicationFailure);
throw conflict;
}
if (attempt >= MAX_PUBLICATION_ATTEMPTS && !shouldContinueRecovery()) {
throw new SQLException("ClickHouse batch remained incomplete after " + attempt + " publication attempts", publicationFailure);
}
pauseBeforeRetry(attempt++, "republishing a ClickHouse batch");
}
}
private RawStatus readStatus(ClickHouseBatchIdentity identity) throws SQLException {
SQLException failure = null;
for (int attempt = 1; attempt <= MAX_RECONCILIATION_ATTEMPTS; attempt++) {
if (Thread.currentThread().isInterrupted()) {
throw interrupted("reconciling a ClickHouse batch", failure);
}
try {
return readRawStatus(identity);
}
catch (SQLException exception) {
if (failure != null && failure != exception) {
exception.addSuppressed(failure);
}
failure = exception;
}
if (attempt < MAX_RECONCILIATION_ATTEMPTS) {
pauseBeforeRetry(attempt, "reconciling a ClickHouse batch");
}
}
throw new SQLException("ClickHouse batch status remained unavailable after " + MAX_RECONCILIATION_ATTEMPTS + " attempts", failure);
}
private RawStatus readRawStatus(ClickHouseBatchIdentity identity) throws SQLException {
String sql = "SELECT count(),uniqExact(tuple(family,rowid)),uniqExact(batch_id),any(toString(batch_id)),uniqExact(batch_ordinal),max(batch_ordinal)"
+ " FROM " + eventTable
+ " WHERE dataset_id=? AND producer_id=? AND producer_sequence=?";
try (Connection connection = jdbc.openConnection(); PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, identity.getDatasetId());
statement.setObject(2, identity.getProducerId());
statement.setLong(3, identity.getProducerSequence());
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse did not return batch reconciliation data");
}
return new RawStatus(resultSet.getLong(1), resultSet.getLong(2), resultSet.getLong(3), resultSet.getString(4), resultSet.getLong(5), resultSet.getLong(6));
}
}
}
private static SQLException conflict(ClickHouseBatchReceipt receipt) {
return new SQLException("ClickHouse batch is partial or conflicting for producer sequence " + receipt.getProducerSequence());
}
private static SQLException interrupted(String operation, SQLException failure) {
InterruptedException interruption = new InterruptedException("Interrupted while " + operation);
SQLException exception = new SQLException(interruption.getMessage(), interruption);
if (failure != null) {
exception.addSuppressed(failure);
}
return exception;
}
private static void pauseBeforeRetry(int attempt, String operation) throws SQLException {
try {
int shift = Math.min(Math.max(0, attempt - 1), 8);
Thread.sleep(Math.min(RETRY_DELAY_MILLIS << shift, MAX_RETRY_DELAY_MILLIS));
}
catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw new SQLException("Interrupted while " + operation, exception);
}
}
private static boolean shouldContinueRecovery() {
return ConfigHandler.serverRunning || ConfigHandler.shutdownDrainRunning;
}
private static final class RawStatus {
private final long count;
private final long logicalRows;
private final long uniqueBatchIds;
private final String batchId;
private final long uniqueOrdinals;
private final long maximumOrdinal;
private RawStatus(long count, long logicalRows, long uniqueBatchIds, String batchId, long uniqueOrdinals, long maximumOrdinal) {
this.count = count;
this.logicalRows = logicalRows;
this.uniqueBatchIds = uniqueBatchIds;
this.batchId = batchId;
this.uniqueOrdinals = uniqueOrdinals;
this.maximumOrdinal = maximumOrdinal;
}
private boolean isEmpty() {
return count == 0;
}
private boolean matches(ClickHouseBatchReceipt receipt) {
long expectedLogicalRows = receipt.getLogicalRowCount();
return count >= expectedLogicalRows
&& count <= receipt.getRowCount()
&& logicalRows == expectedLogicalRows
&& uniqueBatchIds == 1
&& uniqueOrdinals == count
&& maximumOrdinal == receipt.getRowCount() - 1L
&& receipt.getBatchId().toString().equals(batchId);
}
private boolean isRetryablePartial(ClickHouseBatchReceipt receipt) {
return count > 0
&& count < receipt.getRowCount()
&& logicalRows > 0
&& logicalRows <= receipt.getLogicalRowCount()
&& uniqueBatchIds == 1
&& uniqueOrdinals == count
&& maximumOrdinal < receipt.getRowCount()
&& receipt.getBatchId().toString().equals(batchId);
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseBatchReceipt.java
+39
View File
@@ -0,0 +1,39 @@
package net.coreprotect.database.clickhouse;
import java.util.Objects;
import java.util.UUID;
public final class ClickHouseBatchReceipt {
private final long producerSequence;
private final UUID batchId;
private final int rowCount;
private final int logicalRowCount;
ClickHouseBatchReceipt(long producerSequence, UUID batchId, int rowCount, int logicalRowCount) {
if (producerSequence < 1 || rowCount < 0 || logicalRowCount < 0 || logicalRowCount > rowCount) {
throw new IllegalArgumentException("Invalid ClickHouse batch receipt");
}
this.producerSequence = producerSequence;
this.batchId = Objects.requireNonNull(batchId, "batchId");
this.rowCount = rowCount;
this.logicalRowCount = logicalRowCount;
}
public long getProducerSequence() {
return producerSequence;
}
public UUID getBatchId() {
return batchId;
}
public int getRowCount() {
return rowCount;
}
public int getLogicalRowCount() {
return logicalRowCount;
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseConsumerWriteBatch.java
+803
View File
@@ -0,0 +1,803 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.EnumMap;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.UUID;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.consumer.Consumer;
import net.coreprotect.database.ConsumerEntitySpawnUpdates;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.Database;
import net.coreprotect.model.rollback.RollbackUpdateTargets;
import net.coreprotect.utility.ErrorReporter;
public final class ClickHouseConsumerWriteBatch implements ConsumerWriteBatch {
private static final int TARGET_BATCH_ROWS = 100_000;
private final ClickHouseDatabase database;
private final String tablePrefix;
private final String userTable;
private final String usernameLogTable;
private final String databaseLockTable;
private final ClickHouseEntitySpawnUpdates entitySpawnUpdates;
private final EnumMap<ClickHouseFamily, Map<Long, ClickHouseEventPointer>> localPointers = new EnumMap<>(ClickHouseFamily.class);
private final Map<Long, Integer> localBlockActions = new HashMap<>();
private final Map<String, UserRow> usersByName = new HashMap<>();
private final Map<String, UserRow> usersByUuid = new HashMap<>();
private final Map<String, Integer> userAliases = new HashMap<>();
private final Set<String> usernameHistory = new HashSet<>();
private final EnumMap<ReferenceKind, Map<Integer, String>> localReferences = new EnumMap<>(ReferenceKind.class);
private ClickHouseWriteBatch batch;
private Long databaseLockRowId;
private boolean closed;
public ClickHouseConsumerWriteBatch(ClickHouseDatabase database, String prefix) {
this.database = Objects.requireNonNull(database, "database");
String validatedPrefix = prefix == null || prefix.isEmpty() ? "" : ClickHouseIdentifiers.requireIdentifier(prefix, "ClickHouse table prefix");
tablePrefix = validatedPrefix;
userTable = validatedPrefix + ClickHouseFamily.USER.getTableName();
usernameLogTable = validatedPrefix + ClickHouseFamily.USERNAME_LOG.getTableName();
databaseLockTable = validatedPrefix + ClickHouseFamily.DATABASE_LOCK.getTableName();
entitySpawnUpdates = new ClickHouseEntitySpawnUpdates(this, validatedPrefix + "event_data");
}
@Override
public void begin() throws SQLException {
ensureOpen();
if (batch != null) {
throw new IllegalStateException("ClickHouse consumer batch is already active");
}
batch = database.newWriteBatch();
clearBatchState();
entitySpawnUpdates.beginBatch();
Consumer.transacting = true;
}
@Override
public boolean commit() {
ClickHouseWriteBatch current = requireBatch();
boolean published = false;
try {
database.publish(current);
published = true;
try {
publishUserCaches();
}
finally {
entitySpawnUpdates.batchPublished();
}
return true;
}
catch (Exception exception) {
ErrorReporter.report(exception);
return published;
}
finally {
current.close();
batch = null;
clearBatchState();
Consumer.transacting = false;
Consumer.interrupt = false;
if (!published) {
entitySpawnUpdates.batchDiscarded();
}
}
}
@Override
public boolean shouldCommit() {
return batch != null && batch.size() >= TARGET_BATCH_ROWS;
}
@Override
public void rollback() {
if (batch == null) {
return;
}
batch.close();
batch = null;
clearBatchState();
entitySpawnUpdates.batchDiscarded();
Consumer.transacting = false;
Consumer.interrupt = false;
}
@Override
public void executeAtomically(String name, Database.SavepointOperation operation) throws Exception {
Objects.requireNonNull(name, "name");
Objects.requireNonNull(operation, "operation");
ClickHouseWriteBatch current = requireBatch();
ClickHouseWriteBatch.Checkpoint batchCheckpoint = current.checkpoint();
AdapterCheckpoint adapterCheckpoint = checkpoint();
ClickHouseEntitySpawnUpdates.Checkpoint entityCheckpoint = entitySpawnUpdates.checkpoint();
try {
operation.execute();
}
catch (Exception exception) {
try {
current.restore(batchCheckpoint);
restore(adapterCheckpoint);
entitySpawnUpdates.restore(entityCheckpoint);
}
catch (RuntimeException restoreException) {
exception.addSuppressed(restoreException);
rollback();
}
throw exception;
}
}
@Override
public int resolveUserId(String user, String uuid) throws Exception {
requireBatch();
String requiredUser = requireText(user, "user");
String normalizedUuid = normalizeUuid(uuid);
String cacheKey = normalizeName(requiredUser);
Integer cachedId = ConfigHandler.playerIdCache.get(cacheKey);
if (cachedId != null) {
return cachedId;
}
UserRow row = findStagedUser(requiredUser, normalizedUuid);
if (row == null) {
row = findUser(requiredUser, normalizedUuid);
}
if (row == null) {
long rowId = events().addUser(currentTime(), requiredUser, normalizedUuid);
row = new UserRow(toIntId(rowId, "user"), requiredUser, normalizedUuid);
}
stageUser(row);
userAliases.put(cacheKey, row.id);
return row.id;
}
@Override
public void recordUsername(String user, String uuid, int retainHistory, int time) throws Exception {
requireBatch();
String requiredUser = requireText(user, "user");
String requiredUuid = requireText(uuid, "uuid");
if (ConfigHandler.isBlacklisted(requiredUser)) {
return;
}
UserRow row = usersByUuid.get(requiredUuid);
if (row == null) {
row = findUserByUuid(requiredUuid);
}
if (row == null) {
row = findStagedUser(requiredUser, requiredUuid);
}
if (row == null) {
row = findUser(requiredUser, requiredUuid);
}
if (row == null) {
int userId = resolveUserId(requiredUser, requiredUuid);
row = usersByName.get(normalizeName(requiredUser));
if (row == null || row.id != userId) {
throw new SQLException("Unable to stage ClickHouse user " + requiredUser);
}
}
boolean changed = !row.name.equalsIgnoreCase(requiredUser) || !requiredUuid.equals(row.uuid);
if (changed) {
events().addUserVersion(row.id, time, requiredUser, requiredUuid);
unstageUser(row);
row = new UserRow(row.id, requiredUser, requiredUuid);
stageUser(row);
}
if (retainHistory == 1 && (changed || !hasUsernameHistory(requiredUuid, requiredUser))) {
events().addUsernameLog(time, requiredUuid, requiredUser);
usernameHistory.add(historyKey(requiredUuid, requiredUser));
}
}
@Override
public void updateDatabaseLock(int status, int time) throws Exception {
requireBatch();
if (databaseLockRowId == null) {
databaseLockRowId = findDatabaseLockRowId();
}
if (databaseLockRowId == null) {
databaseLockRowId = events().addDatabaseLock(time, status);
}
else {
events().addDatabaseLockVersion(databaseLockRowId, time, status);
}
}
@Override
public void addReference(ReferenceKind kind, int batchCount, int id, String value) throws Exception {
requireBatch();
Objects.requireNonNull(kind, "kind");
String requiredValue = Objects.requireNonNull(value, "value");
Map<Integer, String> references = localReferences.computeIfAbsent(kind, ignored -> new HashMap<>());
for (Map.Entry<Integer, String> reference : references.entrySet()) {
if (reference.getKey() != id && reference.getValue().equals(requiredValue)) {
throw referenceConflict(kind, id, requiredValue, reference.getKey(), reference.getValue());
}
}
String stagedValue = references.get(id);
if (stagedValue != null) {
if (!stagedValue.equals(requiredValue)) {
throw referenceConflict(kind, id, requiredValue, id, stagedValue);
}
return;
}
if (hasCommittedReference(kind, id, requiredValue)) {
references.put(id, requiredValue);
return;
}
references.put(id, requiredValue);
try {
switch (kind) {
case ART:
events().addArtMap(id, requiredValue);
break;
case BLOCK_DATA:
events().addBlockDataMap(id, requiredValue);
break;
case ENTITY:
events().addEntityMap(id, requiredValue);
break;
case MATERIAL:
events().addMaterialMap(id, requiredValue);
break;
case WORLD:
events().addWorld(id, requiredValue);
break;
default:
throw new IllegalArgumentException("Unsupported ClickHouse reference kind: " + kind);
}
}
catch (Exception exception) {
references.remove(id, requiredValue);
throw exception;
}
}
@Override
public void addBlock(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws Exception {
events().addBlock(time, userId, worldId, x, y, z, type, data, meta, blockData, action, rolledBack);
}
@Override
public long addBlockReturningId(int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws Exception {
long rowId = events().addBlock(time, userId, worldId, x, y, z, type, data, meta, blockData, action, rolledBack);
registerPointer(ClickHouseFamily.BLOCK, rowId);
localBlockActions.put(rowId, action);
return rowId;
}
@Override
public int addSkull(int time, String owner, String skin) throws Exception {
return toIntId(events().addSkull(time, owner, skin), "skull");
}
@Override
public void addContainer(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws Exception {
events().addContainer(time, userId, worldId, x, y, z, type, data, amount, metadata, action, rolledBack);
}
@Override
public void addEntityContainer(int batchCount, int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws Exception {
events().addEntityContainer(time, userId, entitySpawnRowId, worldId, x, y, z, type, data, amount, metadata, action, rolledBack);
}
@Override
public void addItem(int batchCount, int time, int userId, int worldId, int x, int y, int z, int type, byte[] data, int amount, int action, int rolledBack) throws Exception {
events().addItem(time, userId, worldId, x, y, z, type, data, amount, action, rolledBack);
}
@Override
public void addChat(int batchCount, long time, int userId, int worldId, int x, int y, int z, String message) throws Exception {
events().addChat(unsignedTime(time), userId, worldId, x, y, z, message);
}
@Override
public void addCommand(int batchCount, long time, int userId, int worldId, int x, int y, int z, String message) throws Exception {
events().addCommand(unsignedTime(time), userId, worldId, x, y, z, message);
}
@Override
public void addSession(int batchCount, int time, int userId, int worldId, int x, int y, int z, int action) throws Exception {
events().addSession(time, userId, worldId, x, y, z, action);
}
@Override
public void addSign(int batchCount, int time, int userId, int worldId, int x, int y, int z, int action, int color, int colorSecondary, int data, int waxed, int face, String[] lines) throws Exception {
events().addSign(time, userId, worldId, x, y, z, action, color, colorSecondary, data, waxed, face, lines);
}
@Override
public int addEntity(int time, byte[] data) throws Exception {
return toIntId(events().addEntity(time, data), "entity");
}
@Override
public int addEntitySpawn(int time, Long blockRowId, Integer killRowId, UUID uuid, int originWorldId, int currentWorldId, double originX, double originY, double originZ, double currentX, double currentY, double currentZ, float yaw, float pitch, byte[] data, int removed) throws Exception {
if (removed != 0 && removed != 1) {
throw new IllegalArgumentException("ClickHouse entity spawn removed state must be zero or one");
}
entitySpawnUpdates.requireAvailable(uuid, killRowId, null);
long rowId = events().addEntitySpawn(time, blockRowId, killRowId, uuid, originWorldId, currentWorldId, originX, originY, originZ, currentX, currentY, currentZ, yaw, pitch, data, removed);
ClickHouseEventPointer pointer = registerPointer(ClickHouseFamily.ENTITY_SPAWN, rowId);
entitySpawnUpdates.register(new ClickHouseEntityState(pointer, blockRowId, killRowId, uuid, currentWorldId, currentX, currentY, currentZ, yaw, pitch, data, removed == 1));
return toIntId(rowId, "entity spawn");
}
@Override
public void linkEntitySpawnBlock(int trackingRowId, long blockRowId) throws Exception {
entitySpawnUpdates.linkBlock(trackingRowId, blockRowId);
}
@Override
public void linkEntitySpawnKill(UUID uuid, int killRowId) throws Exception {
entitySpawnUpdates.linkKill(uuid, killRowId);
}
@Override
public void checkpointEntitySpawn(int trackingRowId, int worldId, double x, double y, double z, float yaw, float pitch) throws Exception {
entitySpawnUpdates.checkpointLocation(trackingRowId, worldId, x, y, z, yaw, pitch);
}
@Override
public void addEntityInteraction(int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int action, byte[] metadata, int rolledBack) throws Exception {
events().addEntityInteraction(time, userId, entitySpawnRowId, worldId, x, y, z, type, action, metadata, rolledBack);
}
@Override
public void updateRolledBack(int target, int rolledBack, List<Long> rowIds) throws Exception {
ClickHouseFamily family = rollbackFamily(target);
Set<Long> unique = validateRowIds(rowIds);
Map<Long, ClickHouseEventPointer> resolved = new LinkedHashMap<>();
Map<Long, ClickHouseEventPointer> local = localPointers.get(family);
List<Long> remote = new ArrayList<>();
for (Long rowId : unique) {
ClickHouseEventPointer pointer = local == null ? null : local.get(rowId);
if (pointer == null) {
remote.add(rowId);
}
else {
resolved.put(rowId, pointer);
}
}
if (!remote.isEmpty()) {
resolved.putAll(database.resolveEventPointers(family, remote));
}
for (Long rowId : unique) {
ClickHouseEventPointer pointer = resolved.get(rowId);
if (pointer == null) {
throw new SQLException("Unable to resolve ClickHouse " + family.getTableName() + " row ID " + rowId);
}
requireBatch().addRollback(pointer, rolledBack);
}
}
@Override
public void updateRolledBackRows(int target, int rolledBack, List<Object[]> rows) throws Exception {
Objects.requireNonNull(rows, "rows");
ClickHouseFamily family = rollbackFamily(target);
if (family == ClickHouseFamily.ENTITY_CONTAINER) {
ConsumerWriteBatch.super.updateRolledBackRows(target, rolledBack, rows);
return;
}
ClickHouseWriteBatch current = requireBatch();
Set<Long> unique = new LinkedHashSet<>();
for (Object[] row : rows) {
if (row == null || row.length <= 10
|| !(row[0] instanceof Long)
|| !(row[1] instanceof Integer)
|| !(row[3] instanceof Integer)
|| !(row[5] instanceof Integer)
|| !(row[10] instanceof Integer)) {
throw new IllegalArgumentException("ClickHouse rollback rows are missing required event coordinates");
}
long rowId = (Long) row[0];
if (unique.add(rowId)) {
current.addRollback(family, rowId, (Integer) row[1], (Integer) row[10], (Integer) row[3], (Integer) row[5], rolledBack);
}
}
}
@Override
public ConsumerEntitySpawnUpdates entitySpawnUpdates() {
requireBatch();
return entitySpawnUpdates;
}
@Override
public void close() {
if (!closed) {
rollback();
entitySpawnUpdates.close();
closed = true;
}
}
Connection openConnection() throws SQLException {
return database.openConnection();
}
ClickHouseDatabase database() {
return database;
}
ClickHouseWriteBatch requireBatch() {
ensureOpen();
if (batch == null) {
throw new IllegalStateException("ClickHouse consumer batch is not active");
}
return batch;
}
boolean updateBlockState(long rowId, int rolledBack, int expectedAction) throws Exception {
Integer action = localBlockActions.get(rowId);
if (action == null) {
prefetchBlockTargets(java.util.Collections.singleton(rowId));
action = localBlockActions.get(rowId);
}
if (action == null || action != expectedAction) {
return false;
}
updateRolledBack(RollbackUpdateTargets.BLOCK, rolledBack, java.util.Collections.singletonList(rowId));
return true;
}
void prefetchBlockTargets(Set<Long> rowIds) throws SQLException {
if (rowIds.isEmpty()) {
return;
}
Map<Long, ClickHouseEventPointer> blockPointers = localPointers.get(ClickHouseFamily.BLOCK);
List<Long> remote = new ArrayList<>();
for (Long rowId : rowIds) {
if (rowId != null && rowId > 0 && (blockPointers == null || !blockPointers.containsKey(rowId) || !localBlockActions.containsKey(rowId))) {
remote.add(rowId);
}
}
if (remote.isEmpty()) {
return;
}
Map<Long, Integer> actions = new HashMap<>();
Map<Long, ClickHouseEventPointer> resolved = database.resolveAvailableBlockTargets(remote, actions);
if (!resolved.isEmpty()) {
Map<Long, ClickHouseEventPointer> pointers = localPointers.computeIfAbsent(ClickHouseFamily.BLOCK, ignored -> new LinkedHashMap<>());
resolved.forEach(pointers::putIfAbsent);
actions.forEach(localBlockActions::putIfAbsent);
}
}
private ClickHouseEventBatch events() {
return requireBatch().events();
}
private ClickHouseEventPointer registerPointer(ClickHouseFamily family, long rowId) {
ClickHouseEventPointer pointer = events().getLastPointer();
if (pointer.getFamily() != family || pointer.getRowId() != rowId) {
throw new IllegalStateException("ClickHouse event pointer does not match the staged fact");
}
localPointers.computeIfAbsent(family, ignored -> new LinkedHashMap<>()).put(rowId, pointer);
return pointer;
}
private UserRow findStagedUser(String user, String uuid) {
UserRow row = usersByName.get(normalizeName(user));
if (row == null && !uuid.isEmpty()) {
row = usersByUuid.get(uuid);
}
return row;
}
private UserRow findUser(String user, String uuid) throws SQLException {
String sql = "SELECT rowid,`user`,uuid FROM " + userTable + " WHERE lowerUTF8(`user`)=lowerUTF8(?)";
if (!uuid.isEmpty()) {
sql += " OR uuid=?";
}
sql += " ORDER BY rowid LIMIT 1";
try (Connection connection = openConnection(); PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setString(1, user);
if (!uuid.isEmpty()) {
statement.setString(2, uuid);
}
return readUniqueUser(statement);
}
}
private UserRow findUserByUuid(String uuid) throws SQLException {
try (Connection connection = openConnection(); PreparedStatement statement = connection.prepareStatement("SELECT rowid,`user`,uuid FROM " + userTable + " WHERE uuid=? ORDER BY rowid LIMIT 1")) {
statement.setString(1, uuid);
return readUniqueUser(statement);
}
}
private UserRow readUniqueUser(PreparedStatement statement) throws SQLException {
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
return null;
}
UserRow row = new UserRow(toIntId(resultSet.getLong(1), "user"), resultSet.getString(2), normalizeUuid(resultSet.getString(3)));
stageUser(row);
return row;
}
}
private boolean hasUsernameHistory(String uuid, String user) throws SQLException {
String key = historyKey(uuid, user);
if (usernameHistory.contains(key)) {
return true;
}
try (Connection connection = openConnection(); PreparedStatement statement = connection.prepareStatement("SELECT 1 FROM " + usernameLogTable + " WHERE uuid=? AND `user`=? LIMIT 1")) {
statement.setString(1, uuid);
statement.setString(2, user);
try (ResultSet resultSet = statement.executeQuery()) {
if (resultSet.next()) {
usernameHistory.add(key);
return true;
}
}
}
return false;
}
private Long findDatabaseLockRowId() throws SQLException {
try (Connection connection = openConnection(); PreparedStatement statement = connection.prepareStatement("SELECT rowid FROM " + databaseLockTable + " ORDER BY rowid LIMIT 2"); ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
return null;
}
long rowId = resultSet.getLong(1);
if (resultSet.next() && resultSet.getLong(1) != rowId) {
throw new SQLException("ClickHouse database lock resolves to multiple rows");
}
return rowId;
}
}
private boolean hasCommittedReference(ReferenceKind kind, int id, String value) throws SQLException {
ClickHouseFamily family = referenceFamily(kind);
String column = referenceValueColumn(kind);
String sql = "SELECT id,`" + column + "` FROM " + tablePrefix + family.getTableName() + " WHERE id=? OR `" + column + "`=?";
boolean found = false;
try (Connection connection = openConnection(); PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setInt(1, id);
statement.setString(2, value);
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
int storedId = resultSet.getInt(1);
String storedValue = resultSet.getString(2);
if (storedId != id || !value.equals(storedValue)) {
throw referenceConflict(kind, id, value, storedId, storedValue);
}
found = true;
}
}
}
return found;
}
private static ClickHouseFamily referenceFamily(ReferenceKind kind) {
switch (kind) {
case ART:
return ClickHouseFamily.ART_MAP;
case BLOCK_DATA:
return ClickHouseFamily.BLOCKDATA_MAP;
case ENTITY:
return ClickHouseFamily.ENTITY_MAP;
case MATERIAL:
return ClickHouseFamily.MATERIAL_MAP;
case WORLD:
return ClickHouseFamily.WORLD;
default:
throw new IllegalArgumentException("Unsupported ClickHouse reference kind: " + kind);
}
}
private static String referenceValueColumn(ReferenceKind kind) {
switch (kind) {
case ART:
return "art";
case BLOCK_DATA:
return "data";
case ENTITY:
return "entity";
case MATERIAL:
return "material";
case WORLD:
return "world";
default:
throw new IllegalArgumentException("Unsupported ClickHouse reference kind: " + kind);
}
}
private static SQLException referenceConflict(ReferenceKind kind, int id, String value, int storedId, String storedValue) {
return new SQLException("ClickHouse " + kind.name().toLowerCase(Locale.ROOT) + " reference " + id + "='" + value + "' conflicts with " + storedId + "='" + storedValue + "'");
}
private void stageUser(UserRow row) {
usersByName.put(normalizeName(row.name), row);
if (!row.uuid.isEmpty()) {
usersByUuid.put(row.uuid, row);
}
}
private static void cacheUser(UserRow row) {
String key = normalizeName(row.name);
ConfigHandler.playerIdCache.put(key, row.id);
ConfigHandler.playerIdCacheReversed.put(row.id, row.name);
if (!row.uuid.isEmpty()) {
ConfigHandler.uuidCache.put(key, row.uuid);
ConfigHandler.uuidCacheReversed.put(row.uuid, row.name);
}
}
private void publishUserCaches() {
for (UserRow row : usersByName.values()) {
cacheUser(row);
}
ConfigHandler.playerIdCache.putAll(userAliases);
}
private void unstageUser(UserRow row) {
usersByName.remove(normalizeName(row.name), row);
if (!row.uuid.isEmpty()) {
usersByUuid.remove(row.uuid, row);
}
}
private AdapterCheckpoint checkpoint() {
EnumMap<ClickHouseFamily, Map<Long, ClickHouseEventPointer>> pointers = new EnumMap<>(ClickHouseFamily.class);
for (Map.Entry<ClickHouseFamily, Map<Long, ClickHouseEventPointer>> entry : localPointers.entrySet()) {
pointers.put(entry.getKey(), new LinkedHashMap<>(entry.getValue()));
}
return new AdapterCheckpoint(pointers, localBlockActions, usersByName, usersByUuid, userAliases, usernameHistory, localReferences, databaseLockRowId);
}
private void restore(AdapterCheckpoint checkpoint) {
localPointers.clear();
for (Map.Entry<ClickHouseFamily, Map<Long, ClickHouseEventPointer>> entry : checkpoint.localPointers.entrySet()) {
localPointers.put(entry.getKey(), new LinkedHashMap<>(entry.getValue()));
}
localBlockActions.clear();
localBlockActions.putAll(checkpoint.localBlockActions);
usersByName.clear();
usersByName.putAll(checkpoint.usersByName);
usersByUuid.clear();
usersByUuid.putAll(checkpoint.usersByUuid);
userAliases.clear();
userAliases.putAll(checkpoint.userAliases);
usernameHistory.clear();
usernameHistory.addAll(checkpoint.usernameHistory);
localReferences.clear();
checkpoint.localReferences.forEach((kind, references) -> localReferences.put(kind, new HashMap<>(references)));
databaseLockRowId = checkpoint.databaseLockRowId;
}
private void clearBatchState() {
localPointers.clear();
localBlockActions.clear();
usersByName.clear();
usersByUuid.clear();
userAliases.clear();
usernameHistory.clear();
localReferences.clear();
databaseLockRowId = null;
}
private void ensureOpen() {
if (closed) {
throw new IllegalStateException("ClickHouse consumer batch is closed");
}
}
private static Set<Long> validateRowIds(List<Long> rowIds) {
Objects.requireNonNull(rowIds, "rowIds");
Set<Long> unique = new LinkedHashSet<>();
for (Long rowId : rowIds) {
if (rowId == null || rowId < 1) {
throw new IllegalArgumentException("ClickHouse rollback row IDs must be positive");
}
unique.add(rowId);
}
return unique;
}
private static ClickHouseFamily rollbackFamily(int target) {
if (target == RollbackUpdateTargets.ENTITY_CONTAINER) {
return ClickHouseFamily.ENTITY_CONTAINER;
}
if (RollbackUpdateTargets.updatesContainerTable(target)) {
return ClickHouseFamily.CONTAINER;
}
if (RollbackUpdateTargets.updatesItemTable(target)) {
return ClickHouseFamily.ITEM;
}
return ClickHouseFamily.BLOCK;
}
private static int toIntId(long rowId, String family) throws SQLException {
if (rowId > Integer.MAX_VALUE) {
throw new SQLException("ClickHouse " + family + " row IDs exceed CoreProtect's signed integer compatibility range");
}
return (int) rowId;
}
private static int unsignedTime(long time) {
if (time < 0 || time > 0xffff_ffffL) {
throw new IllegalArgumentException("ClickHouse timestamps must fit UInt32");
}
return (int) time;
}
private static int currentTime() {
return (int) (System.currentTimeMillis() / 1000L);
}
private static String requireText(String value, String name) {
if (value == null || value.isEmpty()) {
throw new IllegalArgumentException("ClickHouse " + name + " cannot be empty");
}
return value;
}
private static String normalizeUuid(String uuid) {
return uuid == null ? "" : uuid;
}
private static String normalizeName(String user) {
return user.toLowerCase(Locale.ROOT);
}
private static String historyKey(String uuid, String user) {
return uuid + '\0' + user;
}
private static final class UserRow {
private final int id;
private final String name;
private final String uuid;
private UserRow(int id, String name, String uuid) {
this.id = id;
this.name = Objects.requireNonNull(name, "name");
this.uuid = Objects.requireNonNull(uuid, "uuid");
}
}
private static final class AdapterCheckpoint {
private final EnumMap<ClickHouseFamily, Map<Long, ClickHouseEventPointer>> localPointers;
private final Map<Long, Integer> localBlockActions;
private final Map<String, UserRow> usersByName;
private final Map<String, UserRow> usersByUuid;
private final Map<String, Integer> userAliases;
private final Set<String> usernameHistory;
private final EnumMap<ReferenceKind, Map<Integer, String>> localReferences = new EnumMap<>(ReferenceKind.class);
private final Long databaseLockRowId;
private AdapterCheckpoint(EnumMap<ClickHouseFamily, Map<Long, ClickHouseEventPointer>> localPointers, Map<Long, Integer> localBlockActions, Map<String, UserRow> usersByName, Map<String, UserRow> usersByUuid, Map<String, Integer> userAliases, Set<String> usernameHistory, EnumMap<ReferenceKind, Map<Integer, String>> localReferences, Long databaseLockRowId) {
this.localPointers = localPointers;
this.localBlockActions = new HashMap<>(localBlockActions);
this.usersByName = new HashMap<>(usersByName);
this.usersByUuid = new HashMap<>(usersByUuid);
this.userAliases = new HashMap<>(userAliases);
this.usernameHistory = new HashSet<>(usernameHistory);
localReferences.forEach((kind, references) -> this.localReferences.put(kind, new HashMap<>(references)));
this.databaseLockRowId = databaseLockRowId;
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseDatabase.java
+328
View File
@@ -0,0 +1,328 @@
package net.coreprotect.database.clickhouse;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;
import net.coreprotect.config.ConfigHandler;
public final class ClickHouseDatabase implements AutoCloseable {
private static final int MINIMUM_SERVER_MAJOR = 25;
private static final int MINIMUM_SERVER_MINOR = 6;
private final ClickHouseJdbc jdbc;
private final String database;
private final String prefix;
private final UUID datasetId;
private final ClickHouseIdentityAllocator identityAllocator;
private final ClickHouseNativeClient nativeClient;
private final ClickHouseBatchPublisher publisher;
private final ClickHouseRetention retention;
private final ClickHouseTargetResolver targetResolver;
private final ClickHouseWriterRegistration writerRegistration;
private boolean closed;
private ClickHouseDatabase(ClickHouseJdbc jdbc, ClickHouseNativeClient nativeClient, String database, String prefix, UUID datasetId,
ClickHouseIdentityAllocator identityAllocator, ClickHouseWriterRegistration writerRegistration) {
this.jdbc = jdbc;
this.nativeClient = nativeClient;
this.database = database;
this.prefix = prefix;
this.datasetId = datasetId;
this.identityAllocator = identityAllocator;
this.writerRegistration = writerRegistration;
publisher = new ClickHouseBatchPublisher(jdbc, nativeClient, writerRegistration, database, prefix);
retention = new ClickHouseRetention(jdbc, database, prefix, datasetId);
targetResolver = new ClickHouseTargetResolver(jdbc, database, prefix, datasetId);
}
public static ClickHouseDatabase initialize(ClickHouseJdbcConfig config, String prefix) throws SQLException {
return initialize(config, prefix, Paths.get(ConfigHandler.path));
}
public static ClickHouseDatabase initialize(ClickHouseJdbcConfig config, String prefix, Path controlDirectory) throws SQLException {
Objects.requireNonNull(config, "config");
Objects.requireNonNull(controlDirectory, "controlDirectory");
String validatedPrefix = validatePrefix(prefix);
ClickHouseJdbc jdbc = new ClickHouseJdbc(config);
ClickHouseNativeClient nativeClient = null;
ClickHouseWriterRegistration writerRegistration = null;
try {
nativeClient = new ClickHouseNativeClient(config);
List<String> statements = ClickHouseSchema.createStatements(config.getDatabase(), validatedPrefix);
ClickHouseStorageIdentity storageIdentity;
try (Connection connection = jdbc.openConnection()) {
requireServerVersion(connection);
requirePersistentDatabaseIdentity(connection, config.getDatabase());
for (int index = 0; index < ClickHouseSchema.PHYSICAL_TABLE_COUNT; index++) {
jdbc.executeDdl(connection, statements.get(index));
}
ClickHouseSchema.validatePhysicalSchema(connection, config.getDatabase(), validatedPrefix);
storageIdentity = ClickHouseStorageIdentity.load(connection, config.getDatabase(), validatedPrefix);
if (storageIdentity == null) {
ClickHouseSchema.requireUnownedTablesEmpty(connection, config.getDatabase(), validatedPrefix);
storageIdentity = ClickHouseStorageIdentity.loadOrCreate(connection, config.getDatabase(), validatedPrefix);
}
}
writerRegistration = new ClickHouseWriterRegistration(jdbc, config.getDatabase(), validatedPrefix, storageIdentity.getDatasetId(), storageIdentity.getProducerId(), controlDirectory);
writerRegistration.acquire();
ClickHouseHighWaterMarks remoteMarks;
try (Connection connection = jdbc.openConnection()) {
for (int index = ClickHouseSchema.PHYSICAL_TABLE_COUNT; index < statements.size(); index++) {
jdbc.executeDdl(connection, statements.get(index));
}
remoteMarks = ClickHouseStartupReconciler.readRemote(connection, config.getDatabase(), validatedPrefix, storageIdentity.getDatasetId(), storageIdentity.getProducerId());
}
ClickHouseIdentityAllocator identityAllocator = new ClickHouseIdentityAllocator(storageIdentity.getDatasetId(), storageIdentity.getProducerId(), remoteMarks);
ClickHouseDatabase database = new ClickHouseDatabase(jdbc, nativeClient, config.getDatabase(), validatedPrefix, storageIdentity.getDatasetId(), identityAllocator,
writerRegistration);
database.retention.recoverAbandonedTargets();
return database;
}
catch (SQLException | RuntimeException exception) {
try {
if (writerRegistration != null) {
writerRegistration.close();
}
}
catch (SQLException closeException) {
exception.addSuppressed(closeException);
}
finally {
try {
if (nativeClient != null) {
nativeClient.close();
}
}
finally {
jdbc.close();
}
}
throw exception;
}
}
public Connection openConnection() throws SQLException {
ensureOpen();
return jdbc.openConnection();
}
public ClickHouseWriteBatch newWriteBatch() throws SQLException {
ensureOpen();
writerRegistration.ensureOwned();
return new ClickHouseWriteBatch(identityAllocator.nextBatchIdentity(), identityAllocator);
}
public ClickHouseBatchReceipt publish(ClickHouseWriteBatch batch) throws SQLException {
ensureOpen();
return publisher.publish(batch);
}
public Map<Long, ClickHouseEventPointer> resolveEventPointers(ClickHouseFamily family, List<Long> rowIds) throws SQLException {
ensureOpen();
return targetResolver.resolve(family, rowIds);
}
Map<Long, ClickHouseEventPointer> resolveAvailableBlockTargets(List<Long> rowIds, Map<Long, Integer> actions) throws SQLException {
ensureOpen();
return targetResolver.resolveAvailableBlocks(rowIds, actions);
}
public long purge(long startTime, long endTime, int worldId, List<Integer> blockTypes, boolean optimize) throws SQLException {
ensureOpen();
writerRegistration.verifyOwned();
return retention.purge(startTime, endTime, worldId, blockTypes, optimize);
}
public void cancelPurge() {
retention.cancelPurge();
}
UUID getDatasetId() {
return datasetId;
}
public void ensureCoreData(String currentVersion) throws SQLException {
synchronizeCoreData(currentVersion, false);
}
public void updateCoreVersion(String currentVersion) throws SQLException {
synchronizeCoreData(currentVersion, true);
}
private void synchronizeCoreData(String currentVersion, boolean updateVersion) throws SQLException {
ensureOpen();
String requiredVersion = requireCoreVersion(currentVersion);
String versionTable = ClickHouseIdentifiers.qualified(database, prefix + ClickHouseFamily.VERSION.getTableName());
String databaseLockTable = ClickHouseIdentifiers.qualified(database, prefix + ClickHouseFamily.DATABASE_LOCK.getTableName());
CoreData coreData = readCoreData(versionTable, databaseLockTable);
Long versionRowId = coreData.versionRowId;
String storedVersion = coreData.version;
boolean databaseLockExists = coreData.databaseLockExists;
boolean versionSynchronized = versionRowId != null && (!updateVersion || requiredVersion.equals(storedVersion));
if (versionSynchronized && databaseLockExists) {
return;
}
int time = (int) (System.currentTimeMillis() / 1000L);
try (ClickHouseWriteBatch batch = newWriteBatch()) {
if (versionRowId == null) {
batch.events().addVersion(time, requiredVersion);
}
else if (updateVersion && !requiredVersion.equals(storedVersion)) {
batch.events().addVersionRevision(versionRowId, time, requiredVersion);
}
if (!databaseLockExists) {
batch.events().addDatabaseLockVersion(1L, time, 0);
}
publish(batch);
}
}
private CoreData readCoreData(String versionTable, String databaseLockTable) throws SQLException {
Long versionRowId = null;
String storedVersion = null;
boolean databaseLockExists = false;
try (Connection connection = jdbc.openConnection();
PreparedStatement versionStatement = connection.prepareStatement("SELECT rowid,version FROM " + versionTable + " ORDER BY rowid LIMIT 2")) {
try (ResultSet resultSet = versionStatement.executeQuery()) {
if (resultSet.next()) {
versionRowId = resultSet.getLong(1);
storedVersion = resultSet.getString(2);
if (versionRowId != 1L || resultSet.next()) {
throw new SQLException("ClickHouse version data must contain only row ID 1");
}
}
}
try (PreparedStatement lockStatement = connection.prepareStatement("SELECT rowid FROM " + databaseLockTable + " GROUP BY rowid")) {
try (ResultSet resultSet = lockStatement.executeQuery()) {
while (resultSet.next()) {
long rowId = resultSet.getLong(1);
if (rowId != 1L) {
throw new SQLException("ClickHouse database lock data contains an unsupported row ID: " + rowId);
}
databaseLockExists = true;
}
}
}
}
return new CoreData(versionRowId, storedVersion, databaseLockExists);
}
@Override
public synchronized void close() throws SQLException {
if (closed) {
return;
}
closed = true;
try {
writerRegistration.close();
}
finally {
try {
nativeClient.close();
}
finally {
jdbc.close();
}
}
}
private synchronized void ensureOpen() {
if (closed) {
throw new IllegalStateException("ClickHouse database is closed");
}
}
private static final class CoreData {
private final Long versionRowId;
private final String version;
private final boolean databaseLockExists;
private CoreData(Long versionRowId, String version, boolean databaseLockExists) {
this.versionRowId = versionRowId;
this.version = version;
this.databaseLockExists = databaseLockExists;
}
}
private static String requireCoreVersion(String version) {
if (version == null || !version.matches("\\d+\\.\\d+\\.\\d+")) {
throw new IllegalArgumentException("ClickHouse requires a three-part CoreProtect internal database version");
}
return version;
}
private static void requireServerVersion(Connection connection) throws SQLException {
try (PreparedStatement statement = connection.prepareStatement("SELECT version()"); ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse did not return its server version");
}
String version = resultSet.getString(1);
String[] parts = version == null ? new String[0] : version.split("\\.", 3);
int major;
int minor;
try {
major = parts.length > 0 ? Integer.parseInt(parts[0]) : -1;
minor = parts.length > 1 ? Integer.parseInt(parts[1]) : -1;
}
catch (NumberFormatException exception) {
throw new SQLException("Unsupported ClickHouse server version: " + version, exception);
}
if (major < MINIMUM_SERVER_MAJOR || (major == MINIMUM_SERVER_MAJOR && minor < MINIMUM_SERVER_MINOR)) {
throw new SQLException("ClickHouse " + MINIMUM_SERVER_MAJOR + "." + MINIMUM_SERVER_MINOR + " or newer is required; found " + version);
}
}
}
private static void requirePersistentDatabaseIdentity(Connection connection, String database) throws SQLException {
String sql = "SELECT engine,toString(uuid) FROM system.databases WHERE name=? LIMIT 2";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setString(1, database);
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse database " + database + " does not exist");
}
String engine = resultSet.getString(1);
UUID databaseUuid;
try {
databaseUuid = UUID.fromString(resultSet.getString(2));
}
catch (IllegalArgumentException | NullPointerException exception) {
throw new SQLException("ClickHouse database " + database + " did not return a valid persistent UUID", exception);
}
if (resultSet.next()) {
throw new SQLException("ClickHouse database " + database + " is ambiguous");
}
if (databaseUuid.getMostSignificantBits() == 0L && databaseUuid.getLeastSignificantBits() == 0L) {
throw new SQLException("ClickHouse database " + database + " uses engine " + engine
+ ", but CoreProtect requires a database engine with persistent UUID-backed table identities (for example, Atomic)");
}
}
}
}
private static String validatePrefix(String prefix) {
if (prefix == null) {
throw new IllegalArgumentException("ClickHouse table prefix cannot be null");
}
if (!prefix.isEmpty()) {
ClickHouseIdentifiers.requireIdentifier(prefix, "ClickHouse table prefix");
}
return prefix;
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseEntitySpawnUpdates.java
+630
View File
@@ -0,0 +1,630 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.StringJoiner;
import java.util.UUID;
import org.bukkit.Location;
import net.coreprotect.database.ConsumerEntitySpawnUpdates;
import net.coreprotect.database.Database;
import net.coreprotect.database.EntitySpawnUpdateCoordinator;
import net.coreprotect.model.action.LookupActions;
import net.coreprotect.model.entity.EntityContainerRollbackUpdate;
import net.coreprotect.model.entity.EntitySpawnData;
import net.coreprotect.utility.DatabaseUtils;
import net.coreprotect.utility.ErrorReporter;
import net.coreprotect.utility.WorldUtils;
final class ClickHouseEntitySpawnUpdates implements ConsumerEntitySpawnUpdates {
private static final int SELECT_BATCH_SIZE = 500;
private static final String COLUMNS = "rowid,if(block_rowid_present=1,block_rowid,NULL) AS block_rowid,if(kill_rowid_present=1,kill_rowid,NULL) AS kill_rowid,uuid,current_wid,current_x,current_y,current_z,yaw,pitch,"
+ ClickHouseSchema.binary("if(entity_data_present=1,entity_data,NULL)", "data")
+ ",removed,producer_id,producer_sequence,batch_ordinal,time,wid,x AS key_x,z AS key_z";
private final ClickHouseConsumerWriteBatch owner;
private final String eventTable;
private final UUID datasetId;
private final Map<Integer, ClickHouseEntityState> states = new LinkedHashMap<>();
private final Set<Integer> loadedTrackingRowIds = new HashSet<>();
private final Set<UUID> loadedUuids = new HashSet<>();
private final Set<Integer> loadedKillRowIds = new HashSet<>();
private final EntitySpawnUpdateCoordinator coordinator = new EntitySpawnUpdateCoordinator();
private boolean closed;
ClickHouseEntitySpawnUpdates(ClickHouseConsumerWriteBatch owner, String eventTable) {
this.owner = Objects.requireNonNull(owner, "owner");
this.eventTable = Objects.requireNonNull(eventTable, "eventTable");
datasetId = owner.database().getDatasetId();
}
void beginBatch() {
ensureOpen();
clearStateCache();
}
void batchPublished() {
clearStateCache();
}
void batchDiscarded() {
clearStateCache();
}
void register(ClickHouseEntityState state) {
ensureOpen();
cacheState(state, true);
}
@Override
public void prefetch(List<EntitySpawnData> updates) throws Exception {
ensureOpen();
Set<Integer> rowIds = new LinkedHashSet<>();
Set<UUID> uuids = new LinkedHashSet<>();
Set<Integer> killRowIds = new LinkedHashSet<>();
Set<Long> blockRowIds = new LinkedHashSet<>();
for (EntitySpawnData data : updates) {
if (data.getTrackingRowId() > 0) {
rowIds.add(data.getTrackingRowId());
}
if (data.getUuid() != null) {
uuids.add(data.getUuid());
}
if (data.getPreviousUuid() != null) {
uuids.add(data.getPreviousUuid());
}
if (data.getKillRowId() > 0) {
killRowIds.add(data.getKillRowId());
}
EntitySpawnData.Operation operation = data.getOperation();
if (operation == null) {
continue;
}
switch (operation) {
case ROLLBACK:
case RESTORE:
case KILL_ROLLBACK:
case KILL_RESTORE:
addPositive(blockRowIds, data.getBlockRowId());
break;
case COMPOSITE_ROLLBACK:
case COMPOSITE_RESTORE:
addPositive(blockRowIds, data.getBlockRowId());
addPositive(blockRowIds, data.getPairedBlockRowId());
break;
default:
break;
}
}
loadTrackingRows(rowIds);
loadUuids(uuids);
loadKillRowIds(killRowIds);
owner.prefetchBlockTargets(blockRowIds);
}
void linkBlock(int trackingRowId, long blockRowId) throws Exception {
ClickHouseEntityState state = requireState(trackingRowId);
if (state.getBlockRowId() != null) {
throw new SQLException("Entity spawn tracking row already has a block link");
}
append(trackingRowId, state.withBlockRowId(blockRowId));
}
void linkKill(UUID uuid, int killRowId) throws Exception {
ClickHouseEntityState state = findByUuid(Objects.requireNonNull(uuid, "uuid"), null);
if (state == null) {
return;
}
int trackingRowId = toTrackingRowId(state.getPointer().getRowId());
requireAvailable(uuid, killRowId, trackingRowId);
if (Objects.equals(state.getKillRowId(), killRowId)) {
return;
}
append(trackingRowId, state.withKillRowId(killRowId));
}
void requireAvailable(UUID uuid, Integer killRowId, Integer allowedTrackingRowId) throws Exception {
Objects.requireNonNull(uuid, "uuid");
rejectDifferentOwner(findByUuid(uuid, null), allowedTrackingRowId, "entity UUID " + uuid);
if (killRowId != null) {
rejectDifferentOwner(findByKillRowId(killRowId), allowedTrackingRowId, "entity kill row " + killRowId);
}
}
void checkpointLocation(int trackingRowId, int worldId, double x, double y, double z, float yaw, float pitch) throws Exception {
ClickHouseEntityState state = requireState(trackingRowId);
if (state.isRemoved()) {
throw new SQLException("Removed entity spawn tracking row cannot be checkpointed");
}
append(trackingRowId, state.withLocation(worldId, x, y, z, yaw, pitch));
}
@Override
public void apply(EntitySpawnData data) {
Objects.requireNonNull(data, "data");
ensureOpen();
if (!coordinator.begin(data)) {
return;
}
try {
switch (data.getOperation()) {
case VERIFY:
verify(data);
break;
case LOCATION:
updateLocation(data);
break;
case REMOVED:
remove(data);
break;
case REVIVED:
revive(data);
break;
default:
owner.executeAtomically("entity_spawn_transition", () -> applyTransition(data));
break;
}
coordinator.applied(data);
}
catch (PermanentTransitionException exception) {
coordinator.permanentTransitionFailed(data, exception);
ErrorReporter.report(exception);
}
catch (Exception exception) {
coordinator.failed(data);
Database.handleWriteFailure(exception);
}
}
@Override
public void applyCombined(EntityContainerRollbackUpdate update, Database.SavepointOperation rowUpdate) {
Objects.requireNonNull(update, "update");
Objects.requireNonNull(rowUpdate, "rowUpdate");
EntitySpawnData data = update.getTransition();
int trackingRowId = data.getTrackingRowId();
coordinator.beginCombined(update);
try {
owner.executeAtomically("entity_container_transition", () -> {
applyCombinedTransition(data);
rowUpdate.execute();
});
coordinator.combinedApplied(trackingRowId);
}
catch (PermanentTransitionException exception) {
coordinator.permanentCombinedFailed(data, exception);
ErrorReporter.report(exception);
}
catch (Exception exception) {
coordinator.combinedFailed(data);
Database.handleWriteFailure(exception);
}
}
@Override
public void afterCommit(boolean committed) {
coordinator.afterCommit(committed);
clearStateCache();
}
@Override
public void afterRetain() {
coordinator.afterRetain();
clearStateCache();
}
@Override
public void close() {
if (!closed) {
afterCommit(false);
closed = true;
}
}
Checkpoint checkpoint() {
ensureOpen();
return new Checkpoint(this);
}
void restore(Checkpoint checkpoint) {
ensureOpen();
Objects.requireNonNull(checkpoint, "checkpoint");
replace(states, checkpoint.states);
replace(loadedTrackingRowIds, checkpoint.loadedTrackingRowIds);
replace(loadedUuids, checkpoint.loadedUuids);
replace(loadedKillRowIds, checkpoint.loadedKillRowIds);
coordinator.restore(checkpoint.coordinator);
}
private void verify(EntitySpawnData data) throws Exception {
if (findByUuid(data.getUuid(), false) == null) {
coordinator.verificationMissing(data.getUuid());
}
else {
coordinator.verificationFound(data);
}
}
private void updateLocation(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = findByUuid(data.getUuid(), false);
if (state == null) {
coordinator.locationMissing(data.getUuid());
return;
}
append(toTrackingRowId(state.getPointer().getRowId()), withLocation(state, data.getLocation()));
coordinator.locationFound(data);
}
private void remove(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = findByUuid(data.getUuid(), false);
if (state != null) {
append(toTrackingRowId(state.getPointer().getRowId()), withLocation(state, data.getLocation()).withData(null).withRemoved(true));
}
}
private void revive(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = findByUuid(data.getPreviousUuid(), true);
if (state != null) {
requireAvailable(data.getUuid(), null, toTrackingRowId(state.getPointer().getRowId()));
ClickHouseEntityState revived = withLocation(state.withUuid(data.getUuid()), data.getLocation()).withData(null).withRemoved(false);
append(toTrackingRowId(state.getPointer().getRowId()), revived);
coordinator.entityFound(data.getUuid());
}
else if (findByUuid(data.getUuid(), false) == null) {
coordinator.entityMissing(data.getUuid());
}
else {
coordinator.entityFound(data.getUuid());
}
}
private void applyTransition(EntitySpawnData data) throws Exception {
switch (data.getOperation()) {
case ROLLBACK:
applyRollback(data);
break;
case RESTORE:
applyRestore(data);
break;
case KILL_ROLLBACK:
applyKillRollback(data);
break;
case KILL_RESTORE:
applyKillRestore(data);
break;
case COMPOSITE_ROLLBACK:
applyCompositeRollback(data);
break;
case COMPOSITE_RESTORE:
applyCompositeRestore(data);
break;
case CLAIM_RELEASE:
requireState(data.getTrackingRowId());
coordinator.transitionApplied(data.getTrackingRowId());
break;
default:
break;
}
}
private void applyCombinedTransition(EntitySpawnData data) throws Exception {
if (!isTransition(data)) {
throw new PermanentTransitionException("Unsupported combined entity spawn transition " + data.getOperation());
}
applyTransition(data);
}
private void applyRollback(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = requireState(data.getTrackingRowId());
ClickHouseEntityState updated = withLocation(state, data.getLocation()).withData(data.getState()).withKillRowId(null).withRemoved(true);
updateBlockState(data.getBlockRowId(), data.getRolledBack(), LookupActions.ENTITY_SPAWN);
append(data.getTrackingRowId(), updated);
coordinator.transitionApplied(data.getTrackingRowId());
}
private void applyRestore(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = requireState(data.getTrackingRowId());
requireAvailable(data.getUuid(), null, data.getTrackingRowId());
ClickHouseEntityState updated = withLocation(state.withUuid(data.getUuid()), data.getLocation()).withData(null).withKillRowId(null).withRemoved(false);
updateBlockState(data.getBlockRowId(), data.getRolledBack(), LookupActions.ENTITY_SPAWN);
append(data.getTrackingRowId(), updated);
coordinator.entityFound(data.getUuid());
coordinator.transitionApplied(data.getTrackingRowId());
}
private void applyKillRollback(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = requireKillState(data);
requireAvailable(data.getUuid(), null, data.getTrackingRowId());
ClickHouseEntityState updated = withLocation(state.withUuid(data.getUuid()), data.getLocation()).withData(null).withRemoved(false);
updateBlockState(data.getBlockRowId(), data.getRolledBack(), LookupActions.ENTITY_KILL);
append(data.getTrackingRowId(), updated);
coordinator.entityFound(data.getUuid());
coordinator.transitionApplied(data.getTrackingRowId());
}
private void applyKillRestore(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = requireKillState(data);
ClickHouseEntityState updated = withLocation(state, data.getLocation()).withData(null).withRemoved(true);
updateBlockState(data.getBlockRowId(), data.getRolledBack(), LookupActions.ENTITY_KILL);
append(data.getTrackingRowId(), updated);
coordinator.transitionApplied(data.getTrackingRowId());
}
private void applyCompositeRollback(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = requireKillState(data);
ClickHouseEntityState updated = withLocation(state, data.getLocation()).withData(data.getState()).withRemoved(true);
updateBlockState(data.getBlockRowId(), 1, LookupActions.ENTITY_SPAWN);
updateBlockState(data.getPairedBlockRowId(), 1, LookupActions.ENTITY_KILL);
append(data.getTrackingRowId(), updated);
coordinator.transitionApplied(data.getTrackingRowId());
}
private void applyCompositeRestore(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = requireKillState(data);
ClickHouseEntityState updated = state.withData(null).withRemoved(true);
updateBlockState(data.getBlockRowId(), 0, LookupActions.ENTITY_SPAWN);
updateBlockState(data.getPairedBlockRowId(), 0, LookupActions.ENTITY_KILL);
append(data.getTrackingRowId(), updated);
coordinator.transitionApplied(data.getTrackingRowId());
}
private ClickHouseEntityState requireKillState(EntitySpawnData data) throws Exception {
ClickHouseEntityState state = requireState(data.getTrackingRowId());
if (state.getKillRowId() == null || state.getKillRowId() != data.getKillRowId()) {
throw new PermanentTransitionException("Entity spawn tracking row does not match kill row " + data.getKillRowId());
}
return state;
}
private void updateBlockState(long rowId, int rolledBack, int expectedAction) throws Exception {
if (!owner.updateBlockState(rowId, rolledBack, expectedAction)) {
throw new PermanentTransitionException("ClickHouse block row " + rowId + " does not match action " + expectedAction);
}
}
private ClickHouseEntityState requireState(int trackingRowId) throws Exception {
ClickHouseEntityState state = findByRowId(trackingRowId);
if (state == null) {
throw new PermanentTransitionException("Missing entity spawn tracking row " + trackingRowId);
}
return state;
}
private ClickHouseEntityState findByRowId(int trackingRowId) throws Exception {
if (!loadedTrackingRowIds.contains(trackingRowId)) {
loadTrackingRows(Collections.singleton(trackingRowId));
}
return states.get(trackingRowId);
}
private ClickHouseEntityState findByUuid(UUID uuid, Boolean removed) throws Exception {
Objects.requireNonNull(uuid, "uuid");
if (!loadedUuids.contains(uuid)) {
loadUuids(Collections.singleton(uuid));
}
Map<Integer, ClickHouseEntityState> matches = new LinkedHashMap<>();
for (Map.Entry<Integer, ClickHouseEntityState> entry : states.entrySet()) {
if (matches(entry.getValue(), uuid, removed)) {
matches.put(entry.getKey(), entry.getValue());
}
}
if (matches.size() > 1) {
throw new SQLException("ClickHouse entity UUID resolves to multiple tracking rows: " + uuid);
}
return matches.isEmpty() ? null : matches.values().iterator().next();
}
private ClickHouseEntityState findByKillRowId(int killRowId) throws Exception {
if (!loadedKillRowIds.contains(killRowId)) {
loadKillRowIds(Collections.singleton(killRowId));
}
Map<Integer, ClickHouseEntityState> matches = new LinkedHashMap<>();
for (Map.Entry<Integer, ClickHouseEntityState> entry : states.entrySet()) {
if (Objects.equals(entry.getValue().getKillRowId(), killRowId)) {
matches.put(entry.getKey(), entry.getValue());
}
}
if (matches.size() > 1) {
throw new SQLException("ClickHouse kill row resolves to multiple entity spawn rows: " + killRowId);
}
return matches.isEmpty() ? null : matches.values().iterator().next();
}
private void loadTrackingRows(Set<Integer> rowIds) throws Exception {
loadValues("rowid", missing(rowIds, loadedTrackingRowIds), "");
loadedTrackingRowIds.addAll(rowIds);
}
private void loadUuids(Set<UUID> uuids) throws Exception {
List<UUID> missing = missing(uuids, loadedUuids);
List<String> values = new ArrayList<>(missing.size());
for (UUID uuid : missing) {
values.add(uuid.toString());
}
loadValues("uuid", values, "");
loadedUuids.addAll(uuids);
}
private void loadKillRowIds(Set<Integer> killRowIds) throws Exception {
loadValues("kill_rowid", missing(killRowIds, loadedKillRowIds), " AND kill_rowid_present=1");
loadedKillRowIds.addAll(killRowIds);
}
private void loadValues(String column, List<?> values, String additionalFilter) throws Exception {
if (values.isEmpty()) {
return;
}
try (Connection connection = owner.openConnection()) {
for (int offset = 0; offset < values.size(); offset += SELECT_BATCH_SIZE) {
List<?> batch = values.subList(offset, Math.min(offset + SELECT_BATCH_SIZE, values.size()));
StringJoiner placeholders = new StringJoiner(",");
for (int ignored = 0; ignored < batch.size(); ignored++) {
placeholders.add("?");
}
String sql = "SELECT " + COLUMNS + " FROM " + eventTable + " FINAL WHERE dataset_id=? AND family=? AND " + column + " IN(" + placeholders + ")" + additionalFilter;
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.setString(2, ClickHouseFamily.ENTITY_SPAWN.getTableName());
for (int index = 0; index < batch.size(); index++) {
statement.setObject(index + 3, batch.get(index));
}
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
ClickHouseEntityState state = readState(resultSet);
cacheState(state, false);
}
}
}
}
}
}
private static <T> List<T> missing(Set<T> requested, Set<T> loaded) {
List<T> values = new ArrayList<>(requested.size());
for (T value : requested) {
if (!loaded.contains(value)) {
values.add(value);
}
}
return values;
}
private static void addPositive(Set<Long> values, long value) {
if (value > 0) {
values.add(value);
}
}
private static void rejectDifferentOwner(ClickHouseEntityState owner, Integer allowedTrackingRowId, String identity) throws SQLException {
if (owner == null) {
return;
}
int ownerRowId = toTrackingRowId(owner.getPointer().getRowId());
if (allowedTrackingRowId == null || ownerRowId != allowedTrackingRowId) {
throw new SQLException("ClickHouse " + identity + " already belongs to entity spawn row " + ownerRowId);
}
}
private ClickHouseEntityState readState(ResultSet resultSet) throws Exception {
int rowId = resultSet.getInt("rowid");
ClickHouseEventPointer pointer = new ClickHouseEventPointer(datasetId, ClickHouseFamily.ENTITY_SPAWN, UUID.fromString(resultSet.getString("producer_id")), resultSet.getLong("producer_sequence"), resultSet.getInt("batch_ordinal"), rowId, resultSet.getInt("time"), resultSet.getInt("wid"), resultSet.getInt("key_x"), resultSet.getInt("key_z"));
return new ClickHouseEntityState(pointer, nullableLong(resultSet, "block_rowid"), nullableInteger(resultSet, "kill_rowid"), UUID.fromString(resultSet.getString("uuid")), resultSet.getInt("current_wid"), resultSet.getDouble("current_x"), resultSet.getDouble("current_y"), resultSet.getDouble("current_z"), resultSet.getFloat("yaw"), resultSet.getFloat("pitch"), DatabaseUtils.getBytes(resultSet, "data"), resultSet.getInt("removed") == 1);
}
private void append(int trackingRowId, ClickHouseEntityState state) throws SQLException {
if (toTrackingRowId(state.getPointer().getRowId()) != trackingRowId) {
throw new SQLException("ClickHouse entity state does not match tracking row " + trackingRowId);
}
owner.requireBatch().addEntityState(state);
cacheState(state, true);
}
private void cacheState(ClickHouseEntityState state, boolean replace) {
int rowId = toTrackingRowId(state.getPointer().getRowId());
if (replace) {
states.put(rowId, state);
}
else {
states.putIfAbsent(rowId, state);
}
loadedTrackingRowIds.add(rowId);
loadedUuids.add(state.getUuid());
if (state.getKillRowId() != null) {
loadedKillRowIds.add(state.getKillRowId());
}
}
private void clearStateCache() {
states.clear();
loadedTrackingRowIds.clear();
loadedUuids.clear();
loadedKillRowIds.clear();
}
private static boolean isTransition(EntitySpawnData data) {
EntitySpawnData.Operation operation = data.getOperation();
return operation == EntitySpawnData.Operation.ROLLBACK || operation == EntitySpawnData.Operation.RESTORE || operation == EntitySpawnData.Operation.KILL_ROLLBACK || operation == EntitySpawnData.Operation.KILL_RESTORE || operation == EntitySpawnData.Operation.COMPOSITE_ROLLBACK || operation == EntitySpawnData.Operation.COMPOSITE_RESTORE || operation == EntitySpawnData.Operation.CLAIM_RELEASE;
}
private static boolean matches(ClickHouseEntityState state, UUID uuid, Boolean removed) {
return state.getUuid().equals(uuid) && (removed == null || state.isRemoved() == removed);
}
private static ClickHouseEntityState withLocation(ClickHouseEntityState state, Location location) {
Objects.requireNonNull(location, "location");
return state.withLocation(WorldUtils.getWorldId(location.getWorld().getName()), location.getX(), location.getY(), location.getZ(), location.getYaw(), location.getPitch());
}
private static Long nullableLong(ResultSet resultSet, String column) throws SQLException {
Object value = resultSet.getObject(column);
return value == null ? null : ((Number) value).longValue();
}
private static Integer nullableInteger(ResultSet resultSet, String column) throws SQLException {
Object value = resultSet.getObject(column);
return value == null ? null : ((Number) value).intValue();
}
private static int toTrackingRowId(long rowId) {
if (rowId > Integer.MAX_VALUE) {
throw new IllegalStateException("ClickHouse entity spawn row ID exceeds CoreProtect's signed integer compatibility range");
}
return (int) rowId;
}
private void ensureOpen() {
if (closed) {
throw new IllegalStateException("ClickHouse entity spawn updates are closed");
}
}
private static <K, V> void replace(Map<K, V> target, Map<K, V> source) {
target.clear();
target.putAll(source);
}
private static <E> void replace(Set<E> target, Set<E> source) {
target.clear();
target.addAll(source);
}
static final class Checkpoint {
private final Map<Integer, ClickHouseEntityState> states;
private final Set<Integer> loadedTrackingRowIds;
private final Set<UUID> loadedUuids;
private final Set<Integer> loadedKillRowIds;
private final EntitySpawnUpdateCoordinator.Checkpoint coordinator;
private Checkpoint(ClickHouseEntitySpawnUpdates updates) {
states = new LinkedHashMap<>(updates.states);
loadedTrackingRowIds = new HashSet<>(updates.loadedTrackingRowIds);
loadedUuids = new HashSet<>(updates.loadedUuids);
loadedKillRowIds = new HashSet<>(updates.loadedKillRowIds);
coordinator = updates.coordinator.checkpoint();
}
}
private static final class PermanentTransitionException extends SQLException {
private static final long serialVersionUID = 1L;
private PermanentTransitionException(String message) {
super(message);
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseEntityState.java
+115
View File
@@ -0,0 +1,115 @@
package net.coreprotect.database.clickhouse;
import java.util.Objects;
import java.util.UUID;
public final class ClickHouseEntityState {
private final ClickHouseEventPointer pointer;
private final Long blockRowId;
private final Integer killRowId;
private final UUID uuid;
private final int worldId;
private final double x;
private final double y;
private final double z;
private final float yaw;
private final float pitch;
private final byte[] data;
private final boolean removed;
public ClickHouseEntityState(ClickHouseEventPointer pointer, Long blockRowId, Integer killRowId, UUID uuid, int worldId, double x, double y, double z, float yaw, float pitch, byte[] data, boolean removed) {
this.pointer = Objects.requireNonNull(pointer, "pointer");
if (pointer.getFamily() != ClickHouseFamily.ENTITY_SPAWN) {
throw new IllegalArgumentException("ClickHouse entity state must target an entity_spawn event");
}
this.blockRowId = blockRowId;
this.killRowId = killRowId;
this.uuid = Objects.requireNonNull(uuid, "uuid");
this.worldId = worldId;
this.x = x;
this.y = y;
this.z = z;
this.yaw = yaw;
this.pitch = pitch;
this.data = data == null ? null : data.clone();
this.removed = removed;
}
public ClickHouseEventPointer getPointer() {
return pointer;
}
public Long getBlockRowId() {
return blockRowId;
}
public Integer getKillRowId() {
return killRowId;
}
public UUID getUuid() {
return uuid;
}
public int getWorldId() {
return worldId;
}
public double getX() {
return x;
}
public double getY() {
return y;
}
public double getZ() {
return z;
}
public float getYaw() {
return yaw;
}
public float getPitch() {
return pitch;
}
public byte[] getData() {
return data == null ? null : data.clone();
}
public boolean isRemoved() {
return removed;
}
ClickHouseEntityState withBlockRowId(Long value) {
return copy(value, killRowId, uuid, worldId, x, y, z, yaw, pitch, data, removed);
}
ClickHouseEntityState withKillRowId(Integer value) {
return copy(blockRowId, value, uuid, worldId, x, y, z, yaw, pitch, data, removed);
}
ClickHouseEntityState withUuid(UUID value) {
return copy(blockRowId, killRowId, value, worldId, x, y, z, yaw, pitch, data, removed);
}
ClickHouseEntityState withLocation(int nextWorldId, double nextX, double nextY, double nextZ, float nextYaw, float nextPitch) {
return copy(blockRowId, killRowId, uuid, nextWorldId, nextX, nextY, nextZ, nextYaw, nextPitch, data, removed);
}
ClickHouseEntityState withData(byte[] value) {
return copy(blockRowId, killRowId, uuid, worldId, x, y, z, yaw, pitch, value, removed);
}
ClickHouseEntityState withRemoved(boolean value) {
return copy(blockRowId, killRowId, uuid, worldId, x, y, z, yaw, pitch, data, value);
}
private ClickHouseEntityState copy(Long nextBlockRowId, Integer nextKillRowId, UUID nextUuid, int nextWorldId, double nextX, double nextY, double nextZ, float nextYaw, float nextPitch, byte[] nextData, boolean nextRemoved) {
return new ClickHouseEntityState(pointer, nextBlockRowId, nextKillRowId, nextUuid, nextWorldId, nextX, nextY, nextZ, nextYaw, nextPitch, nextData, nextRemoved);
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseEventBatch.java
+420
View File
@@ -0,0 +1,420 @@
package net.coreprotect.database.clickhouse;
import java.io.InputStream;
import java.sql.SQLException;
import java.util.EnumMap;
import java.util.HashSet;
import java.util.Objects;
import java.util.UUID;
public final class ClickHouseEventBatch implements AutoCloseable {
private final ClickHouseBatchIdentity identity;
private final ClickHouseRowIdAllocator rowIdAllocator;
private final ClickHouseRowBinaryBuffer rows = new ClickHouseRowBinaryBuffer(ClickHouseSchema.EVENT_COLUMNS, ClickHouseSchema.EVENT_COLUMN_TYPES);
private final EnumMap<ClickHouseFamily, HashSet<Long>> versionRowIds = new EnumMap<>(ClickHouseFamily.class);
private int eventCount;
private int duplicateVersionCount;
private int currentTime;
private int currentWorldId;
private int currentX;
private int currentZ;
private ClickHouseEventPointer lastPointer;
private boolean sealed;
private boolean closed;
ClickHouseEventBatch(ClickHouseBatchIdentity identity, ClickHouseRowIdAllocator rowIdAllocator) {
this.identity = Objects.requireNonNull(identity, "identity");
this.rowIdAllocator = Objects.requireNonNull(rowIdAllocator, "rowIdAllocator");
}
public long addArtMap(int id, String art) throws SQLException {
return addNamedMap(ClickHouseFamily.ART_MAP, id, art);
}
public long addBlock(int time, int userId, int worldId, int x, int y, int z, int type, int data, byte[] meta, byte[] blockData, int action, int rolledBack) throws SQLException {
long rowId = beginRow(ClickHouseFamily.BLOCK, time);
setLocation(userId, worldId, x, y, z);
set("type", type);
set("data", (long) data);
setBinary("meta", meta);
setBinary("blockdata", blockData);
set("action", action);
set("rolled_back", rolledBack);
commitRow(ClickHouseFamily.BLOCK, rowId);
return rowId;
}
public long addChat(int time, int userId, int worldId, int x, int y, int z, String message) throws SQLException {
return addMessage(ClickHouseFamily.CHAT, time, userId, worldId, x, y, z, message);
}
public long addCommand(int time, int userId, int worldId, int x, int y, int z, String message) throws SQLException {
return addMessage(ClickHouseFamily.COMMAND, time, userId, worldId, x, y, z, message);
}
public long addContainer(int time, int userId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws SQLException {
return addContainer(ClickHouseFamily.CONTAINER, time, userId, 0, worldId, x, y, z, type, data, amount, metadata, action, rolledBack);
}
public long addEntityContainer(int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws SQLException {
return addContainer(ClickHouseFamily.ENTITY_CONTAINER, time, userId, entitySpawnRowId, worldId, x, y, z, type, data, amount, metadata, action, rolledBack);
}
public long addEntityInteraction(int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int action, byte[] metadata, int rolledBack) throws SQLException {
long rowId = beginRow(ClickHouseFamily.ENTITY_INTERACTION, time);
setLocation(userId, worldId, x, y, z);
set("entity_spawn_rowid", (long) entitySpawnRowId);
set("type", type);
set("action", action);
setBinary("metadata", metadata);
set("rolled_back", rolledBack);
commitRow(ClickHouseFamily.ENTITY_INTERACTION, rowId);
return rowId;
}
public long addItem(int time, int userId, int worldId, int x, int y, int z, int type, byte[] data, int amount, int action, int rolledBack) throws SQLException {
long rowId = beginRow(ClickHouseFamily.ITEM, time);
setLocation(userId, worldId, x, y, z);
set("type", type);
setBinary("payload", data);
set("amount", amount);
set("action", action);
set("rolled_back", rolledBack);
commitRow(ClickHouseFamily.ITEM, rowId);
return rowId;
}
public long addDatabaseLock(int time, int status) throws SQLException {
long rowId = beginRow(ClickHouseFamily.DATABASE_LOCK, time);
set("status", status);
set("database_lock_time", time);
commitRow(ClickHouseFamily.DATABASE_LOCK, rowId);
return rowId;
}
public long addDatabaseLockVersion(long rowId, int time, int status) throws SQLException {
beginRow(ClickHouseFamily.DATABASE_LOCK, rowId, time);
set("status", status);
set("database_lock_time", time);
commitRow(ClickHouseFamily.DATABASE_LOCK, rowId);
return rowId;
}
public long addEntity(int time, byte[] data) throws SQLException {
long rowId = beginRow(ClickHouseFamily.ENTITY, time);
setBinary("payload", data);
commitRow(ClickHouseFamily.ENTITY, rowId);
return rowId;
}
public long addEntitySpawn(int time, Long blockRowId, Integer killRowId, UUID uuid, int worldId, int currentWorldId, double originX, double originY, double originZ, double currentX, double currentY, double currentZ, float yaw, float pitch, byte[] data, int removed) throws SQLException {
long rowId = beginRow(ClickHouseFamily.ENTITY_SPAWN, time);
set("block_rowid", blockRowId);
set("kill_rowid", killRowId == null ? null : killRowId.longValue());
set("block_rowid_present", blockRowId == null ? 0 : 1);
set("kill_rowid_present", killRowId == null ? 0 : 1);
set("uuid", Objects.requireNonNull(uuid, "uuid").toString());
set("wid", worldId);
set("x", (int) Math.floor(originX));
set("z", (int) Math.floor(originZ));
set("current_wid", currentWorldId);
set("origin_x", originX);
set("origin_y", originY);
set("origin_z", originZ);
set("current_x", currentX);
set("current_y", currentY);
set("current_z", currentZ);
set("yaw", yaw);
set("pitch", pitch);
setBinary("entity_data", data);
set("entity_data_present", data == null ? 0 : 1);
set("removed", removed);
commitRow(ClickHouseFamily.ENTITY_SPAWN, rowId);
return rowId;
}
public long addEntityMap(int id, String entity) throws SQLException {
return addNamedMap(ClickHouseFamily.ENTITY_MAP, id, entity);
}
public long addMaterialMap(int id, String material) throws SQLException {
return addNamedMap(ClickHouseFamily.MATERIAL_MAP, id, material);
}
public long addBlockDataMap(int id, String blockData) throws SQLException {
long rowId = beginRow(ClickHouseFamily.BLOCKDATA_MAP, 0);
set("id", id);
set("text", Objects.requireNonNull(blockData, "blockData"));
commitRow(ClickHouseFamily.BLOCKDATA_MAP, rowId);
return rowId;
}
public long addSession(int time, int userId, int worldId, int x, int y, int z, int action) throws SQLException {
long rowId = beginRow(ClickHouseFamily.SESSION, time);
setLocation(userId, worldId, x, y, z);
set("action", action);
commitRow(ClickHouseFamily.SESSION, rowId);
return rowId;
}
public long addSign(int time, int userId, int worldId, int x, int y, int z, int action, int color, int colorSecondary, int data, int waxed, int face, String... lines) throws SQLException {
if (lines == null || lines.length != 8) {
throw new IllegalArgumentException("ClickHouse sign events require exactly eight lines");
}
long rowId = beginRow(ClickHouseFamily.SIGN, time);
setLocation(userId, worldId, x, y, z);
set("action", action);
set("color", color);
set("color_secondary", colorSecondary);
set("sign_data", data);
set("waxed", waxed);
set("face", face);
for (int index = 0; index < lines.length; index++) {
set("line_" + (index + 1), Objects.requireNonNull(lines[index], "sign line"));
}
commitRow(ClickHouseFamily.SIGN, rowId);
return rowId;
}
public long addSkull(int time, String owner, String skin) throws SQLException {
long rowId = beginRow(ClickHouseFamily.SKULL, time);
set("name", owner == null ? "" : owner);
set("text", skin == null ? "" : skin);
commitRow(ClickHouseFamily.SKULL, rowId);
return rowId;
}
public long addUser(int time, String userName, String uuid) throws SQLException {
long rowId = beginRow(ClickHouseFamily.USER, time);
setUser(userName, uuid);
commitRow(ClickHouseFamily.USER, rowId);
return rowId;
}
public long addUserVersion(long rowId, int time, String userName, String uuid) throws SQLException {
beginRow(ClickHouseFamily.USER, rowId, time);
setUser(userName, uuid);
commitRow(ClickHouseFamily.USER, rowId);
return rowId;
}
public long addUsernameLog(int time, String uuid, String userName) throws SQLException {
long rowId = beginRow(ClickHouseFamily.USERNAME_LOG, time);
setUser(userName, uuid);
commitRow(ClickHouseFamily.USERNAME_LOG, rowId);
return rowId;
}
public long addVersion(int time, String version) throws SQLException {
long rowId = beginRow(ClickHouseFamily.VERSION, time);
set("version", Objects.requireNonNull(version, "version"));
commitRow(ClickHouseFamily.VERSION, rowId);
return rowId;
}
public long addVersionRevision(long rowId, int time, String version) throws SQLException {
beginRow(ClickHouseFamily.VERSION, rowId, time);
set("version", Objects.requireNonNull(version, "version"));
commitRow(ClickHouseFamily.VERSION, rowId);
return rowId;
}
public long addWorld(int id, String world) throws SQLException {
return addNamedMap(ClickHouseFamily.WORLD, id, world);
}
public int size() {
return eventCount;
}
int logicalSize() {
return eventCount - duplicateVersionCount;
}
void seal(ClickHouseStateBatch state) throws SQLException {
if (sealed) {
return;
}
ensureWritable();
Objects.requireNonNull(state, "state").appendTo(rows, eventCount);
rows.seal();
sealed = true;
}
InputStream openRows() {
if (!sealed) {
throw new IllegalStateException("ClickHouse event batch is not sealed");
}
return rows.openStream();
}
ClickHouseEventPointer getLastPointer() {
if (lastPointer == null) {
throw new IllegalStateException("ClickHouse event batch has no rows");
}
return lastPointer;
}
Checkpoint checkpoint() {
ensureWritable();
return new Checkpoint(rows.checkpoint(), eventCount, duplicateVersionCount, versionRowIds, lastPointer);
}
void restore(Checkpoint checkpoint) {
ensureWritable();
Objects.requireNonNull(checkpoint, "checkpoint");
rows.restore(checkpoint.bufferSize);
eventCount = checkpoint.eventCount;
duplicateVersionCount = checkpoint.duplicateVersionCount;
versionRowIds.clear();
checkpoint.versionRowIds.forEach((family, rowIds) -> versionRowIds.put(family, new HashSet<>(rowIds)));
lastPointer = checkpoint.lastPointer;
}
@Override
public void close() {
closed = true;
sealed = true;
rows.close();
}
private long addNamedMap(ClickHouseFamily family, int id, String name) throws SQLException {
long rowId = beginRow(family, 0);
set("id", id);
set("name", Objects.requireNonNull(name, family.getTableName()));
commitRow(family, rowId);
return rowId;
}
private long addMessage(ClickHouseFamily family, int time, int userId, int worldId, int x, int y, int z, String message) throws SQLException {
long rowId = beginRow(family, time);
setLocation(userId, worldId, x, y, z);
set("message", Objects.requireNonNull(message, "message"));
commitRow(family, rowId);
return rowId;
}
private long addContainer(ClickHouseFamily family, int time, int userId, int entitySpawnRowId, int worldId, int x, int y, int z, int type, int data, int amount, byte[] metadata, int action, int rolledBack) throws SQLException {
long rowId = beginRow(family, time);
setLocation(userId, worldId, x, y, z);
set("entity_spawn_rowid", (long) entitySpawnRowId);
set("type", type);
set("data", (long) data);
set("amount", amount);
setBinary("metadata", metadata);
set("action", action);
set("rolled_back", rolledBack);
commitRow(family, rowId);
return rowId;
}
private long beginRow(ClickHouseFamily family, int time) {
return beginRow(family, rowIdAllocator.nextRowId(family), time);
}
private long beginRow(ClickHouseFamily family, long rowId, int time) {
ensureWritable();
Objects.requireNonNull(family, "family");
if (eventCount == Integer.MAX_VALUE) {
throw new IllegalStateException("ClickHouse event batch exceeds the supported row count");
}
if (rowId < 1) {
throw new IllegalArgumentException("ClickHouse compatibility row IDs must be positive");
}
rows.beginRow();
currentTime = time;
currentWorldId = 0;
currentX = 0;
currentZ = 0;
set("dataset_id", identity.getDatasetId());
set("producer_id", identity.getProducerId());
set("producer_sequence", identity.getProducerSequence());
set("batch_id", identity.getBatchId());
set("batch_ordinal", eventCount);
set("family", family.getTableName());
set("rowid", rowId);
set("time", time);
return rowId;
}
private void setLocation(int userId, int worldId, int x, int y, int z) {
set("user_id", userId);
set("wid", worldId);
set("x", x);
set("y", y);
set("z", z);
}
private void setUser(String userName, String uuid) {
set("user_name", Objects.requireNonNull(userName, "userName"));
set("uuid", uuid == null ? "" : uuid);
}
private void setBinary(String column, byte[] value) {
set(column, value);
}
private void set(String column, Object value) {
rows.set(column, value);
if (value == null) {
return;
}
switch (column) {
case "time":
currentTime = ((Number) value).intValue();
break;
case "wid":
currentWorldId = ((Number) value).intValue();
break;
case "x":
currentX = ((Number) value).intValue();
break;
case "z":
currentZ = ((Number) value).intValue();
break;
default:
break;
}
}
private void commitRow(ClickHouseFamily family, long rowId) throws SQLException {
rows.commitRow(family.getTableName() + " event");
if (isVersionedFamily(family) && !versionRowIds.computeIfAbsent(family, ignored -> new HashSet<>()).add(rowId)) {
duplicateVersionCount++;
}
lastPointer = new ClickHouseEventPointer(identity.getDatasetId(), family, identity.getProducerId(), identity.getProducerSequence(), eventCount, rowId, currentTime, currentWorldId, currentX, currentZ);
eventCount++;
}
private static boolean isVersionedFamily(ClickHouseFamily family) {
return family == ClickHouseFamily.DATABASE_LOCK || family == ClickHouseFamily.USER || family == ClickHouseFamily.VERSION;
}
private void ensureWritable() {
if (closed) {
throw new IllegalStateException("ClickHouse event batch is closed");
}
if (sealed) {
throw new IllegalStateException("ClickHouse event batch is already sealed");
}
}
static final class Checkpoint {
private final int bufferSize;
private final int eventCount;
private final int duplicateVersionCount;
private final EnumMap<ClickHouseFamily, HashSet<Long>> versionRowIds = new EnumMap<>(ClickHouseFamily.class);
private final ClickHouseEventPointer lastPointer;
private Checkpoint(int bufferSize, int eventCount, int duplicateVersionCount, EnumMap<ClickHouseFamily, HashSet<Long>> versionRowIds, ClickHouseEventPointer lastPointer) {
this.bufferSize = bufferSize;
this.eventCount = eventCount;
this.duplicateVersionCount = duplicateVersionCount;
versionRowIds.forEach((family, rowIds) -> this.versionRowIds.put(family, new HashSet<>(rowIds)));
this.lastPointer = lastPointer;
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseEventPointer.java
+101
View File
@@ -0,0 +1,101 @@
package net.coreprotect.database.clickhouse;
import java.util.Objects;
import java.util.UUID;
public final class ClickHouseEventPointer {
private final ClickHouseFamily family;
private final UUID datasetId;
private final UUID producerId;
private final long producerSequence;
private final int batchOrdinal;
private final long rowId;
private final int time;
private final int worldId;
private final int x;
private final int z;
public ClickHouseEventPointer(UUID datasetId, ClickHouseFamily family, UUID producerId, long producerSequence, int batchOrdinal, long rowId, int time, int worldId, int x, int z) {
this.datasetId = Objects.requireNonNull(datasetId, "datasetId");
this.family = Objects.requireNonNull(family, "family");
this.producerId = Objects.requireNonNull(producerId, "producerId");
if (producerSequence < 1 || batchOrdinal < 0 || rowId < 1) {
throw new IllegalArgumentException("ClickHouse event pointers require positive sequence/row IDs and a non-negative ordinal");
}
this.producerSequence = producerSequence;
this.batchOrdinal = batchOrdinal;
this.rowId = rowId;
this.time = time;
this.worldId = worldId;
this.x = x;
this.z = z;
}
public ClickHouseFamily getFamily() {
return family;
}
public UUID getDatasetId() {
return datasetId;
}
public UUID getProducerId() {
return producerId;
}
public long getProducerSequence() {
return producerSequence;
}
public int getBatchOrdinal() {
return batchOrdinal;
}
public long getRowId() {
return rowId;
}
public int getTime() {
return time;
}
public int getWorldId() {
return worldId;
}
public int getX() {
return x;
}
public int getZ() {
return z;
}
@Override
public boolean equals(Object object) {
if (this == object) {
return true;
}
if (!(object instanceof ClickHouseEventPointer)) {
return false;
}
ClickHouseEventPointer pointer = (ClickHouseEventPointer) object;
return producerSequence == pointer.producerSequence
&& batchOrdinal == pointer.batchOrdinal
&& rowId == pointer.rowId
&& time == pointer.time
&& worldId == pointer.worldId
&& x == pointer.x
&& z == pointer.z
&& family == pointer.family
&& datasetId.equals(pointer.datasetId)
&& producerId.equals(pointer.producerId);
}
@Override
public int hashCode() {
return Objects.hash(datasetId, family, producerId, producerSequence, batchOrdinal, rowId, time, worldId, x, z);
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseFamily.java
+69
View File
@@ -0,0 +1,69 @@
package net.coreprotect.database.clickhouse;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.Map;
import net.coreprotect.database.PurgePolicy;
public enum ClickHouseFamily {
ART_MAP("art_map"),
BLOCK("block"),
CHAT("chat"),
COMMAND("command"),
CONTAINER("container"),
ENTITY_CONTAINER("entity_container"),
ENTITY_INTERACTION("entity_interaction"),
ITEM("item"),
DATABASE_LOCK("database_lock"),
ENTITY("entity"),
ENTITY_SPAWN("entity_spawn"),
ENTITY_MAP("entity_map"),
MATERIAL_MAP("material_map"),
BLOCKDATA_MAP("blockdata_map"),
SESSION("session"),
SIGN("sign"),
SKULL("skull"),
USER("user"),
USERNAME_LOG("username_log"),
VERSION("version"),
WORLD("world");
private static final Map<String, ClickHouseFamily> BY_TABLE_NAME;
static {
Map<String, ClickHouseFamily> families = new LinkedHashMap<>();
for (ClickHouseFamily family : values()) {
families.put(family.tableName, family);
}
BY_TABLE_NAME = Collections.unmodifiableMap(families);
}
private final String tableName;
ClickHouseFamily(String tableName) {
this.tableName = tableName;
}
public String getTableName() {
return tableName;
}
boolean isPurgeable() {
return PurgePolicy.isPurgeable(tableName);
}
boolean isWorldScoped() {
return PurgePolicy.isWorldScoped(tableName);
}
public static ClickHouseFamily fromTableName(String tableName) {
ClickHouseFamily family = BY_TABLE_NAME.get(tableName);
if (family == null) {
throw new IllegalArgumentException("Unknown ClickHouse event family: " + tableName);
}
return family;
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseHighWaterMarks.java
+39
View File
@@ -0,0 +1,39 @@
package net.coreprotect.database.clickhouse;
import java.util.Collections;
import java.util.EnumMap;
import java.util.Map;
import java.util.Objects;
public final class ClickHouseHighWaterMarks {
private final long producerSequence;
private final Map<ClickHouseFamily, Long> compatibilityRowIds;
public ClickHouseHighWaterMarks(long producerSequence, Map<ClickHouseFamily, Long> compatibilityRowIds) {
if (producerSequence < 0) {
throw new IllegalArgumentException("Producer sequence cannot be negative");
}
Objects.requireNonNull(compatibilityRowIds, "compatibilityRowIds");
EnumMap<ClickHouseFamily, Long> validatedRowIds = new EnumMap<>(ClickHouseFamily.class);
for (Map.Entry<ClickHouseFamily, Long> entry : compatibilityRowIds.entrySet()) {
ClickHouseFamily family = Objects.requireNonNull(entry.getKey(), "event family");
Long rowId = Objects.requireNonNull(entry.getValue(), "compatibility row ID");
if (rowId < 0) {
throw new IllegalArgumentException("Compatibility row ID cannot be negative");
}
validatedRowIds.put(family, rowId);
}
this.producerSequence = producerSequence;
this.compatibilityRowIds = Collections.unmodifiableMap(validatedRowIds);
}
public long getProducerSequence() {
return producerSequence;
}
public long getCompatibilityRowId(ClickHouseFamily family) {
return compatibilityRowIds.getOrDefault(Objects.requireNonNull(family, "family"), 0L);
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseIdentifiers.java
+27
View File
@@ -0,0 +1,27 @@
package net.coreprotect.database.clickhouse;
import java.util.regex.Pattern;
final class ClickHouseIdentifiers {
private static final Pattern IDENTIFIER = Pattern.compile("[A-Za-z_][A-Za-z0-9_]*");
private ClickHouseIdentifiers() {
throw new IllegalStateException("Utility class");
}
static String requireIdentifier(String value, String name) {
if (value == null || !IDENTIFIER.matcher(value).matches()) {
throw new IllegalArgumentException(name + " must be a simple ClickHouse identifier");
}
return value;
}
static String quote(String value, String name) {
return "`" + requireIdentifier(value, name) + "`";
}
static String qualified(String database, String table) {
return quote(database, "ClickHouse database") + "." + quote(table, "ClickHouse table");
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseIdentityAllocator.java
+52
View File
@@ -0,0 +1,52 @@
package net.coreprotect.database.clickhouse;
import java.util.EnumMap;
import java.util.Objects;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicLong;
public final class ClickHouseIdentityAllocator implements ClickHouseRowIdAllocator {
private final UUID datasetId;
private final UUID producerId;
private final AtomicLong producerSequence;
private final EnumMap<ClickHouseFamily, AtomicLong> rowIds = new EnumMap<>(ClickHouseFamily.class);
public ClickHouseIdentityAllocator(UUID datasetId, UUID producerId, ClickHouseHighWaterMarks highWaterMarks) {
this.datasetId = Objects.requireNonNull(datasetId, "datasetId");
this.producerId = Objects.requireNonNull(producerId, "producerId");
Objects.requireNonNull(highWaterMarks, "highWaterMarks");
producerSequence = new AtomicLong(highWaterMarks.getProducerSequence());
for (ClickHouseFamily family : ClickHouseFamily.values()) {
rowIds.put(family, new AtomicLong(highWaterMarks.getCompatibilityRowId(family)));
}
}
public ClickHouseBatchIdentity nextBatchIdentity() {
long sequence = increment(producerSequence, "producer sequence");
return ClickHouseBatchIdentity.create(datasetId, producerId, sequence);
}
UUID getDatasetId() {
return datasetId;
}
@Override
public long nextRowId(ClickHouseFamily family) {
Objects.requireNonNull(family, "family");
long rowId = increment(rowIds.get(family), family.getTableName() + " row ID");
if (family != ClickHouseFamily.BLOCK && rowId > Integer.MAX_VALUE) {
throw new IllegalStateException("ClickHouse " + family.getTableName() + " row IDs exceed CoreProtect's signed 32-bit compatibility range");
}
return rowId;
}
private static long increment(AtomicLong value, String name) {
long next = value.incrementAndGet();
if (next < 1) {
throw new IllegalStateException("ClickHouse " + name + " exhausted its signed 64-bit range");
}
return next;
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseJdbc.java
+89
View File
@@ -0,0 +1,89 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Locale;
import java.util.Objects;
import java.util.Set;
import com.clickhouse.jdbc.DataSourceImpl;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
public final class ClickHouseJdbc implements AutoCloseable {
private static final Set<String> DDL_OPERATIONS = Set.of("ALTER", "CREATE", "DROP", "RENAME", "TRUNCATE");
private final HikariDataSource dataSource;
public ClickHouseJdbc(ClickHouseJdbcConfig config) {
Objects.requireNonNull(config, "config");
HikariConfig pool = new HikariConfig();
pool.setDataSource(new HikariClickHouseDataSource(config));
pool.setPoolName("CoreProtect-ClickHouse");
pool.setMinimumIdle(0);
pool.setMaximumPoolSize(10);
pool.setConnectionTimeout(10_000L);
pool.setIdleTimeout(60_000L);
pool.setInitializationFailTimeout(-1L);
dataSource = new HikariDataSource(pool);
}
public Connection openConnection() throws SQLException {
return dataSource.getConnection();
}
public void executeDdl(String ddl) throws SQLException {
try (Connection connection = openConnection()) {
executeDdl(connection, ddl);
}
}
public void executeDdl(Connection connection, String ddl) throws SQLException {
Objects.requireNonNull(connection, "connection");
String statementSql = requireSingleDdl(ddl);
try (Statement statement = connection.createStatement()) {
statement.execute(statementSql);
}
}
@Override
public void close() {
dataSource.close();
}
private static String requireSingleDdl(String ddl) {
if (ddl == null) {
throw new IllegalArgumentException("ClickHouse DDL cannot be null");
}
String normalized = ddl.trim();
if (normalized.isEmpty()) {
throw new IllegalArgumentException("ClickHouse DDL cannot be empty");
}
if (normalized.indexOf(';') >= 0) {
throw new IllegalArgumentException("Execute one ClickHouse DDL statement at a time without a semicolon");
}
int separator = 0;
while (separator < normalized.length() && !Character.isWhitespace(normalized.charAt(separator))) {
separator++;
}
String operation = normalized.substring(0, separator).toUpperCase(Locale.ROOT);
if (!DDL_OPERATIONS.contains(operation)) {
throw new IllegalArgumentException("Unsupported ClickHouse DDL operation: " + operation);
}
return normalized;
}
private static final class HikariClickHouseDataSource extends DataSourceImpl {
private HikariClickHouseDataSource(ClickHouseJdbcConfig config) {
super(config.getJdbcUrl(), config.getProperties());
}
@Override
public void setLoginTimeout(int seconds) {
// ClickHouse enforces connection_timeout directly.
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseJdbcConfig.java
+116
View File
@@ -0,0 +1,116 @@
package net.coreprotect.database.clickhouse;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Properties;
public final class ClickHouseJdbcConfig {
private static final String SERVER_SETTING_PREFIX = "clickhouse_setting_";
private final String host;
private final int port;
private final String database;
private final String username;
private final String password;
private final boolean tls;
private final String jdbcUrl;
private final Properties properties;
public ClickHouseJdbcConfig(String host, int port, String database, String username, String password, boolean tls) {
String validatedHost = validateHost(host);
if (port < 1 || port > 65535) {
throw new IllegalArgumentException("ClickHouse port must be between 1 and 65535");
}
String validatedDatabase = ClickHouseIdentifiers.requireIdentifier(database, "ClickHouse database");
if (username == null || username.isEmpty()) {
throw new IllegalArgumentException("ClickHouse username cannot be empty");
}
if (password == null) {
throw new IllegalArgumentException("ClickHouse password cannot be null");
}
this.host = validatedHost;
this.port = port;
this.database = validatedDatabase;
this.username = username;
this.password = password;
this.tls = tls;
jdbcUrl = createJdbcUrl(validatedHost, port, validatedDatabase, tls);
properties = new Properties();
properties.setProperty("user", username);
properties.setProperty("password", password);
properties.setProperty("connection_timeout", "10000");
properties.setProperty("socket_timeout", "300000");
properties.setProperty("retry", "0");
properties.setProperty(SERVER_SETTING_PREFIX + "async_insert", "0");
properties.setProperty(SERVER_SETTING_PREFIX + "do_not_merge_across_partitions_select_final", "1");
properties.setProperty(SERVER_SETTING_PREFIX + "wait_end_of_query", "1");
properties.setProperty("ssl", Boolean.toString(tls));
properties.setProperty("jdbc_ignore_unsupported_values", "false");
}
public String getJdbcUrl() {
return jdbcUrl;
}
public String getDatabase() {
return database;
}
String getEndpoint() {
return (tls ? "https" : "http") + "://" + formatHost(host) + ":" + port;
}
String getUsername() {
return username;
}
String getPassword() {
return password;
}
public ClickHouseJdbcConfig withDatabase(String database) {
return new ClickHouseJdbcConfig(host, port, database, username, password, tls);
}
public Properties getProperties() {
Properties copy = new Properties();
copy.putAll(properties);
return copy;
}
private static String validateHost(String host) {
if (host == null || host.isEmpty() || !host.equals(host.trim())) {
throw new IllegalArgumentException("ClickHouse host cannot be empty or contain surrounding whitespace");
}
String normalized = host;
if (host.startsWith("[") && host.endsWith("]")) {
normalized = host.substring(1, host.length() - 1);
}
try {
URI uri = new URI("http", null, normalized, 1, null, null, null);
if (uri.getHost() == null) {
throw new IllegalArgumentException("Invalid ClickHouse host: " + host);
}
}
catch (URISyntaxException exception) {
throw new IllegalArgumentException("Invalid ClickHouse host: " + host, exception);
}
return normalized;
}
private static String createJdbcUrl(String host, int port, String database, boolean tls) {
try {
URI endpoint = new URI(tls ? "https" : "http", null, host, port, "/" + database, null, null);
return "jdbc:clickhouse:" + endpoint.toASCIIString();
}
catch (URISyntaxException exception) {
throw new IllegalArgumentException("Invalid ClickHouse connection configuration", exception);
}
}
private static String formatHost(String host) {
return host.indexOf(':') >= 0 ? "[" + host + "]" : host;
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseNativeClient.java
+73
View File
@@ -0,0 +1,73 @@
package net.coreprotect.database.clickhouse;
import java.io.IOException;
import java.io.InputStream;
import java.sql.SQLException;
import java.time.temporal.ChronoUnit;
import java.util.List;
import java.util.Objects;
import java.util.concurrent.ExecutionException;
import com.clickhouse.client.api.Client;
import com.clickhouse.client.api.ClientFaultCause;
import com.clickhouse.client.api.insert.InsertResponse;
import com.clickhouse.client.api.insert.InsertSettings;
import com.clickhouse.data.ClickHouseFormat;
final class ClickHouseNativeClient implements AutoCloseable {
private final Client client;
ClickHouseNativeClient(ClickHouseJdbcConfig config) {
Objects.requireNonNull(config, "config");
client = new Client.Builder()
.addEndpoint(config.getEndpoint())
.setUsername(config.getUsername())
.setPassword(config.getPassword())
.setDefaultDatabase(config.getDatabase())
.setConnectTimeout(10, ChronoUnit.SECONDS)
.setSocketTimeout(5, ChronoUnit.MINUTES)
.setMaxRetries(0)
.retryOnFailures(ClientFaultCause.None)
.useAsyncRequests(false)
.compressClientRequest(true)
.serverSetting("async_insert", "0")
.serverSetting("max_insert_block_size", "1000000")
.serverSetting("max_insert_block_size_bytes", "0")
.serverSetting("wait_end_of_query", "1")
.build();
}
void insert(String table, List<String> columns, InputStream rows, ClickHouseBatchIdentity identity, String streamName) throws SQLException {
Objects.requireNonNull(columns, "columns");
Objects.requireNonNull(rows, "rows");
Objects.requireNonNull(identity, "identity");
String stream = ClickHouseIdentifiers.requireIdentifier(streamName, "ClickHouse batch stream");
InsertSettings settings = new InsertSettings()
.setDeduplicationToken(identity.getDeduplicationToken() + ":" + stream)
.setQueryId(identity.getBatchId() + "-" + stream)
.compressClientRequest(true)
.serverSetting("async_insert", "0")
.serverSetting("max_insert_block_size", "1000000")
.serverSetting("max_insert_block_size_bytes", "0")
.serverSetting("wait_end_of_query", "1");
try (InputStream input = rows; InsertResponse ignored = client.insert(table, columns, input, ClickHouseFormat.RowBinary, settings).get()) {
}
catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw new SQLException("Interrupted while publishing a ClickHouse batch", exception);
}
catch (IOException exception) {
throw new SQLException("Failed to close ClickHouse batch publication stream", exception);
}
catch (ExecutionException | RuntimeException exception) {
Throwable cause = exception instanceof ExecutionException && exception.getCause() != null ? exception.getCause() : exception;
throw new SQLException("ClickHouse batch publication failed", cause);
}
}
@Override
public void close() {
client.close();
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseRetention.java
+376
View File
@@ -0,0 +1,376 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import java.util.StringJoiner;
import java.util.UUID;
import net.coreprotect.database.PurgePolicy;
import net.coreprotect.model.action.LookupActions;
final class ClickHouseRetention {
private static final String MUTATION_SETTINGS = " SETTINGS mutations_sync=2,allow_nondeterministic_mutations=1";
private static final String PURGEABLE_FAMILIES = purgeableFamilies();
private final ClickHouseJdbc jdbc;
private final UUID datasetId;
private final String eventTable;
private final String highWaterTable;
private final String prefix;
private final String database;
private final String databaseName;
private final String eventTableName;
private volatile Connection activePurgeConnection;
private volatile boolean purgeCancellationRequested;
ClickHouseRetention(ClickHouseJdbc jdbc, String database, String prefix, UUID datasetId) {
this.jdbc = Objects.requireNonNull(jdbc, "jdbc");
this.datasetId = Objects.requireNonNull(datasetId, "datasetId");
databaseName = ClickHouseIdentifiers.requireIdentifier(database, "ClickHouse database");
this.database = ClickHouseIdentifiers.quote(databaseName, "ClickHouse database");
this.prefix = prefix == null || prefix.isEmpty() ? "" : ClickHouseIdentifiers.requireIdentifier(prefix, "ClickHouse table prefix");
eventTableName = this.prefix + "event_data";
eventTable = table("event_data");
highWaterTable = table("retention_high_water");
}
synchronized void recoverAbandonedTargets() throws SQLException {
try (Connection connection = jdbc.openConnection()) {
cleanupAbandonedTargets(connection);
}
}
void cancelPurge() {
purgeCancellationRequested = true;
Connection connection = activePurgeConnection;
if (connection == null) {
return;
}
try {
connection.abort(Runnable::run);
}
catch (Exception ignored) {
try {
connection.close();
}
catch (Exception closeIgnored) {
}
}
}
synchronized long purge(long startTime, long endTime, int worldId, List<Integer> blockTypes, boolean optimize) throws SQLException {
if (startTime < 0 || endTime <= startTime || endTime > 0xffff_ffffL) {
throw new IllegalArgumentException("Invalid ClickHouse purge time range");
}
if (worldId < 0) {
throw new IllegalArgumentException("ClickHouse purge world ID cannot be negative");
}
List<Integer> restrictedBlockTypes = blockTypes == null ? Collections.emptyList() : new ArrayList<>(blockTypes);
boolean directTimePurge = worldId == 0 && restrictedBlockTypes.isEmpty();
String targetSuffix = "purge_targets_" + UUID.randomUUID().toString().replace("-", "");
String targetTableName = prefix + targetSuffix;
String targetTable = table(targetSuffix);
requirePurgeNotCancelled();
try (Connection connection = jdbc.openConnection()) {
activePurgeConnection = connection;
requirePurgeNotCancelled();
cleanupAbandonedTargets(connection);
snapshotHighWaterMarks(connection);
boolean completed = false;
Throwable failure = null;
try {
createTargets(connection, targetTable);
long removed;
if (directTimePurge) {
removed = countPrimaryRows(connection, startTime, endTime);
dropCoveredPartitions(connection, startTime, endTime);
deletePrimaryRows(connection, startTime, endTime);
}
else {
selectPrimaryTargets(connection, targetTable, startTime, endTime, worldId, restrictedBlockTypes);
removed = countTargets(connection, targetTable);
deleteTargets(connection, targetTable);
}
removed += cleanupEntityData(connection, targetTable);
if (optimize) {
try (Statement statement = connection.createStatement()) {
statement.execute("OPTIMIZE TABLE " + eventTable + " FINAL");
}
}
completed = true;
return removed;
}
catch (SQLException | RuntimeException | Error exception) {
failure = exception;
throw exception;
}
finally {
try {
if (completed || !hasPendingMutation(connection, targetTableName)) {
dropTargets(connection, targetTable);
}
}
catch (SQLException cleanupFailure) {
if (failure == null) {
throw cleanupFailure;
}
failure.addSuppressed(cleanupFailure);
}
}
}
finally {
activePurgeConnection = null;
}
}
private void requirePurgeNotCancelled() throws SQLException {
if (purgeCancellationRequested) {
throw new SQLException("ClickHouse purge cancelled during shutdown");
}
}
private void cleanupAbandonedTargets(Connection connection) throws SQLException {
String targetPrefix = prefix + "purge_targets_";
String sql = "SELECT name FROM system.tables WHERE database=? AND startsWith(name,?)";
List<String> candidates = new ArrayList<>();
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setString(1, databaseName);
statement.setString(2, targetPrefix);
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
String tableName = resultSet.getString(1);
String suffix = tableName.substring(targetPrefix.length());
if (suffix.matches("[0-9a-f]{32}")) {
candidates.add(tableName);
}
}
}
}
for (String candidate : candidates) {
if (!hasPendingMutation(connection, candidate)) {
dropTargets(connection, database + "." + ClickHouseIdentifiers.quote(candidate, "ClickHouse purge target table"));
}
}
}
private boolean hasPendingMutation(Connection connection, String targetTableName) throws SQLException {
String sql = "SELECT 1 FROM system.mutations WHERE database=? AND table=? AND is_done=0 AND position(command,?)>0 LIMIT 1";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setString(1, databaseName);
statement.setString(2, eventTableName);
statement.setString(3, targetTableName);
try (ResultSet resultSet = statement.executeQuery()) {
return resultSet.next();
}
}
}
private void snapshotHighWaterMarks(Connection connection) throws SQLException {
String sql = "INSERT INTO " + highWaterTable
+ " (dataset_id,producer_id,producer_sequence,family,rowid,recorded_at)"
+ " SELECT dataset_id,any(producer_id),max(producer_sequence),family,max(rowid),now64(3, 'UTC')"
+ " FROM " + eventTable + " WHERE dataset_id=? GROUP BY dataset_id,family";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.execute();
}
}
private void createTargets(Connection connection, String targetTable) throws SQLException {
execute(connection, "CREATE TABLE " + targetTable
+ " (family LowCardinality(String),rowid UInt64) ENGINE = MergeTree ORDER BY (family,rowid)"
+ " SETTINGS fsync_after_insert=1,fsync_part_directory=1");
}
private void dropTargets(Connection connection, String targetTable) throws SQLException {
execute(connection, "DROP TABLE IF EXISTS " + targetTable);
}
private void clearTargets(Connection connection, String targetTable) throws SQLException {
execute(connection, "TRUNCATE TABLE " + targetTable);
}
private void selectPrimaryTargets(Connection connection, String targetTable, long startTime, long endTime, int worldId, List<Integer> blockTypes) throws SQLException {
for (ClickHouseFamily family : ClickHouseFamily.values()) {
if (!family.isPurgeable()) {
continue;
}
if (!blockTypes.isEmpty() && !PurgePolicy.supportsBlockRestriction(family.getTableName())) {
continue;
}
if (worldId > 0 && !family.isWorldScoped()) {
continue;
}
StringBuilder sql = new StringBuilder("INSERT INTO ").append(targetTable)
.append(" SELECT '").append(family.getTableName()).append("',rowid FROM ").append(table(family.getTableName()))
.append(" WHERE time>=? AND time<?");
List<Integer> integerParameters = new ArrayList<>();
if (PurgePolicy.supportsBlockRestriction(family.getTableName()) && !blockTypes.isEmpty()) {
sql.append(" AND action NOT IN(").append(LookupActions.ENTITY_KILL).append(',').append(LookupActions.ENTITY_SPAWN).append(") AND type IN(");
StringJoiner placeholders = new StringJoiner(",");
for (Integer blockType : blockTypes) {
placeholders.add("?");
integerParameters.add(Objects.requireNonNull(blockType, "block type"));
}
sql.append(placeholders).append(')');
}
if (worldId > 0) {
if (family == ClickHouseFamily.ENTITY_CONTAINER || family == ClickHouseFamily.ENTITY_INTERACTION) {
sql.append(" AND (wid=? OR entity_spawn_rowid IN(SELECT rowid FROM ").append(table("entity_spawn")).append(" WHERE current_wid=?))");
integerParameters.add(worldId);
integerParameters.add(worldId);
}
else {
sql.append(" AND wid=?");
integerParameters.add(worldId);
}
}
try (PreparedStatement statement = connection.prepareStatement(sql.toString())) {
statement.setLong(1, startTime);
statement.setLong(2, endTime);
for (int index = 0; index < integerParameters.size(); index++) {
statement.setInt(index + 3, integerParameters.get(index));
}
statement.execute();
}
}
}
private long countPrimaryRows(Connection connection, long startTime, long endTime) throws SQLException {
long count = 0;
for (ClickHouseFamily family : ClickHouseFamily.values()) {
if (!family.isPurgeable()) {
continue;
}
String sql = "SELECT count() FROM " + table(family.getTableName()) + " WHERE time>=? AND time<?";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setLong(1, startTime);
statement.setLong(2, endTime);
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse did not return a purge row count");
}
count += resultSet.getLong(1);
}
}
}
return count;
}
private void deletePrimaryRows(Connection connection, long startTime, long endTime) throws SQLException {
String sql = "ALTER TABLE " + eventTable + " DELETE WHERE dataset_id=toUUID('" + datasetId + "')"
+ " AND family IN(" + PURGEABLE_FAMILIES + ") AND time>=" + startTime + " AND time<" + endTime
+ MUTATION_SETTINGS;
execute(connection, sql);
}
private long cleanupEntityData(Connection connection, String targetTable) throws SQLException {
clearMissingReference(connection, "block_rowid", "block_rowid_present", "block");
clearMissingReference(connection, "kill_rowid", "kill_rowid_present", "entity");
clearTargets(connection, targetTable);
selectOrphanChildren(connection, targetTable, ClickHouseFamily.ENTITY_CONTAINER);
selectOrphanChildren(connection, targetTable, ClickHouseFamily.ENTITY_INTERACTION);
long removed = countTargets(connection, targetTable);
deleteTargets(connection, targetTable);
clearTargets(connection, targetTable);
String orphanSpawns = "INSERT INTO " + targetTable
+ " SELECT 'entity_spawn',rowid FROM " + table("entity_spawn")
+ " WHERE removed=1 AND block_rowid IS NULL AND kill_rowid IS NULL"
+ " AND rowid NOT IN(SELECT assumeNotNull(entity_spawn_rowid) FROM " + table("entity_container") + " WHERE entity_spawn_rowid IS NOT NULL)"
+ " AND rowid NOT IN(SELECT assumeNotNull(entity_spawn_rowid) FROM " + table("entity_interaction") + " WHERE entity_spawn_rowid IS NOT NULL)";
execute(connection, orphanSpawns);
removed += countTargets(connection, targetTable);
deleteTargets(connection, targetTable);
return removed;
}
private void clearMissingReference(Connection connection, String valueColumn, String presenceColumn, String referencedFamily) throws SQLException {
String sql = "ALTER TABLE " + eventTable + " UPDATE " + valueColumn + "=NULL," + presenceColumn + "=0"
+ " WHERE dataset_id=toUUID('" + datasetId + "') AND family='entity_spawn' AND " + presenceColumn + "=1"
+ " AND " + valueColumn + " IS NOT NULL AND " + valueColumn + " NOT IN(SELECT rowid FROM " + table(referencedFamily) + ")"
+ MUTATION_SETTINGS;
execute(connection, sql);
}
private void selectOrphanChildren(Connection connection, String targetTable, ClickHouseFamily family) throws SQLException {
String sql = "INSERT INTO " + targetTable + " SELECT '" + family.getTableName() + "',rowid FROM " + table(family.getTableName())
+ " WHERE entity_spawn_rowid IS NULL OR assumeNotNull(entity_spawn_rowid) NOT IN(SELECT rowid FROM " + table("entity_spawn") + ")";
execute(connection, sql);
}
private long countTargets(Connection connection, String targetTable) throws SQLException {
try (Statement statement = connection.createStatement(); ResultSet resultSet = statement.executeQuery("SELECT count() FROM " + targetTable)) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse did not return a purge target count");
}
return resultSet.getLong(1);
}
}
private void deleteTargets(Connection connection, String targetTable) throws SQLException {
if (countTargets(connection, targetTable) == 0) {
return;
}
String sql = "ALTER TABLE " + eventTable + " DELETE WHERE dataset_id=toUUID('" + datasetId + "')"
+ " AND (family,rowid) IN(SELECT family,rowid FROM " + targetTable + ")" + MUTATION_SETTINGS;
execute(connection, sql);
}
private void dropCoveredPartitions(Connection connection, long startTime, long endTime) throws SQLException {
String sql = "SELECT _partition_id,min(time),max(time) FROM " + eventTable
+ " WHERE dataset_id=? AND family IN(" + PURGEABLE_FAMILIES + ") GROUP BY _partition_id";
List<String> partitions = new ArrayList<>();
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
String partition = resultSet.getString(1);
long minimum = resultSet.getLong(2);
long maximum = resultSet.getLong(3);
if (!"0".equals(partition) && partition != null && partition.matches("[0-9]+") && minimum >= startTime && maximum < endTime) {
partitions.add(partition);
}
}
}
}
for (String partition : partitions) {
execute(connection, "ALTER TABLE " + eventTable + " DROP PARTITION ID '" + partition + "'");
}
}
private void execute(Connection connection, String sql) throws SQLException {
try (Statement statement = connection.createStatement()) {
statement.execute(sql);
}
}
private String table(String suffix) {
return database + "." + ClickHouseIdentifiers.quote(prefix + suffix, "ClickHouse table");
}
private static String purgeableFamilies() {
StringJoiner families = new StringJoiner(",");
for (ClickHouseFamily family : ClickHouseFamily.values()) {
if (family.isPurgeable()) {
families.add("'" + family.getTableName() + "'");
}
}
return families.toString();
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseRowBinaryBuffer.java
+253
View File
@@ -0,0 +1,253 @@
package net.coreprotect.database.clickhouse;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.sql.SQLException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import com.clickhouse.data.format.BinaryStreamUtils;
final class ClickHouseRowBinaryBuffer implements AutoCloseable {
private static final UUID ZERO_UUID = new UUID(0, 0);
private final RowBuffer rows = new RowBuffer();
private final Map<String, Integer> columnIndexes = new HashMap<>();
private final String[] types;
private final Object[] defaults;
private final Object[] values;
private boolean sealed;
private boolean failed;
private boolean rowStarted;
ClickHouseRowBinaryBuffer(List<String> columns, List<String> types) {
if (columns.size() != types.size() || columns.isEmpty()) {
throw new IllegalArgumentException("ClickHouse RowBinary columns and types must have the same non-zero size");
}
this.types = types.toArray(new String[0]);
defaults = new Object[types.size()];
values = new Object[types.size()];
for (int index = 0; index < columns.size(); index++) {
String column = columns.get(index);
if (columnIndexes.put(column, index) != null) {
throw new IllegalArgumentException("Duplicate ClickHouse RowBinary column: " + column);
}
defaults[index] = defaultValue(types.get(index));
}
}
void beginRow() {
ensureWritable();
if (rowStarted) {
throw new IllegalStateException("ClickHouse RowBinary row is already active");
}
System.arraycopy(defaults, 0, values, 0, defaults.length);
rowStarted = true;
}
void set(String column, Object value) {
ensureWritable();
requireActiveRow();
Integer index = columnIndexes.get(column);
if (index == null) {
throw new IllegalArgumentException("Unknown ClickHouse RowBinary column: " + column);
}
values[index] = value;
}
void commitRow(String description) throws SQLException {
ensureWritable();
requireActiveRow();
try {
for (int index = 0; index < values.length; index++) {
writeValue(types[index], values[index]);
}
rowStarted = false;
}
catch (IOException | RuntimeException exception) {
failed = true;
throw new SQLException("Failed to encode ClickHouse " + description, exception);
}
}
void seal() {
ensureWritable();
if (rowStarted) {
throw new IllegalStateException("ClickHouse RowBinary row is incomplete");
}
sealed = true;
}
InputStream openStream() {
if (!sealed) {
throw new IllegalStateException("ClickHouse RowBinary buffer is not sealed");
}
return rows.openStream();
}
int checkpoint() {
ensureWritable();
if (rowStarted) {
throw new IllegalStateException("Cannot checkpoint an incomplete ClickHouse RowBinary row");
}
return rows.size();
}
void restore(int checkpoint) {
if (sealed) {
throw new IllegalStateException("ClickHouse RowBinary buffer is sealed");
}
if (checkpoint < 0 || checkpoint > rows.size()) {
throw new IllegalArgumentException("Invalid ClickHouse RowBinary checkpoint");
}
rows.truncate(checkpoint);
failed = false;
rowStarted = false;
}
@Override
public void close() {
sealed = true;
rows.reset();
}
private void ensureWritable() {
if (sealed) {
throw new IllegalStateException("ClickHouse RowBinary buffer is sealed");
}
if (failed) {
throw new IllegalStateException("ClickHouse RowBinary buffer is invalid after an encoding failure");
}
}
private void requireActiveRow() {
if (!rowStarted) {
throw new IllegalStateException("ClickHouse RowBinary row is not active");
}
}
private void writeValue(String declaredType, Object value) throws IOException {
String type = declaredType;
if (type.startsWith("Nullable(") && type.endsWith(")")) {
if (value == null) {
BinaryStreamUtils.writeNull(rows);
return;
}
BinaryStreamUtils.writeNonNull(rows);
type = type.substring(9, type.length() - 1);
}
else if (value == null) {
throw new IllegalArgumentException("Non-nullable ClickHouse value cannot be null");
}
if (type.startsWith("LowCardinality(") && type.endsWith(")")) {
type = type.substring(15, type.length() - 1);
}
switch (type) {
case "UUID":
BinaryStreamUtils.writeUuid(rows, (UUID) value);
return;
case "String":
if (value instanceof byte[]) {
BinaryStreamUtils.writeString(rows, (byte[]) value);
}
else {
BinaryStreamUtils.writeString(rows, (String) value);
}
return;
case "UInt8":
BinaryStreamUtils.writeUnsignedInt8(rows, unsignedInt8(value));
return;
case "UInt32":
BinaryStreamUtils.writeUnsignedInt32(rows, unsignedInt32(value));
return;
case "UInt64":
BinaryStreamUtils.writeUnsignedInt64(rows, unsignedInt64(value));
return;
case "Int32":
BinaryStreamUtils.writeInt32(rows, int32(value));
return;
case "Int64":
BinaryStreamUtils.writeInt64(rows, ((Number) value).longValue());
return;
case "Float32":
BinaryStreamUtils.writeFloat32(rows, ((Number) value).floatValue());
return;
case "Float64":
BinaryStreamUtils.writeFloat64(rows, ((Number) value).doubleValue());
return;
default:
throw new IllegalArgumentException("Unsupported ClickHouse RowBinary type: " + declaredType);
}
}
private static int unsignedInt8(Object value) {
long number = ((Number) value).longValue();
if (number < 0 || number > 0xffL) {
throw new IllegalArgumentException("ClickHouse UInt8 value is out of range: " + number);
}
return (int) number;
}
private static long unsignedInt32(Object value) {
long number = value instanceof Integer ? Integer.toUnsignedLong((Integer) value) : ((Number) value).longValue();
if (number < 0 || number > 0xffff_ffffL) {
throw new IllegalArgumentException("ClickHouse UInt32 value is out of range: " + number);
}
return number;
}
private static long unsignedInt64(Object value) {
long number = ((Number) value).longValue();
if (number < 0) {
throw new IllegalArgumentException("ClickHouse UInt64 value is out of range: " + number);
}
return number;
}
private static int int32(Object value) {
long number = ((Number) value).longValue();
if (number < Integer.MIN_VALUE || number > Integer.MAX_VALUE) {
throw new IllegalArgumentException("ClickHouse Int32 value is out of range: " + number);
}
return (int) number;
}
private static Object defaultValue(String type) {
if (type.startsWith("Nullable(")) {
return null;
}
if (type.equals("UUID")) {
return ZERO_UUID;
}
if (type.contains("String")) {
return "";
}
if (type.equals("Float32")) {
return 0.0f;
}
if (type.equals("Float64")) {
return 0.0d;
}
if (type.equals("UInt64") || type.equals("Int64")) {
return 0L;
}
return 0;
}
private static final class RowBuffer extends ByteArrayOutputStream {
private InputStream openStream() {
return new ByteArrayInputStream(buf, 0, count);
}
private void truncate(int size) {
count = size;
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseRowIdAllocator.java
+8
View File
@@ -0,0 +1,8 @@
package net.coreprotect.database.clickhouse;
@FunctionalInterface
public interface ClickHouseRowIdAllocator {
long nextRowId(ClickHouseFamily family);
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseSchema.java
+487
View File
@@ -0,0 +1,487 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.StringJoiner;
public final class ClickHouseSchema {
static final int VERSION = 1;
private static final String INTEGER_CODEC = " CODEC(Delta, ZSTD(3))";
private static final String VALUE_CODEC = " CODEC(ZSTD(3))";
private static final String PURGEABLE_FAMILIES = purgeableFamilies();
private static final String EVENT_PARTITION_KEY = "if(family IN (" + PURGEABLE_FAMILIES + "),toYYYYMM(toDateTime(time,'UTC')),0)";
private static final String EVENT_SORTING_KEY = "(dataset_id,family,wid,x,z,if(family IN ('database_lock','user','version'),0,time),rowid)";
private static final String[][] EVENT_DATA_SKIPPING_INDEX_DEFINITIONS = {
{ "producer_sequence_idx", "producer_sequence", "minmax", "1" },
{ "rowid_idx", "rowid", "bloom_filter(0.01)", "1" },
{ "entity_uuid_idx", "uuid", "bloom_filter(0.01)", "1" },
{ "entity_kill_rowid_idx", "kill_rowid", "bloom_filter(0.01)", "1" }
};
private static final String[][] STORAGE_METADATA_COLUMN_DEFINITIONS = {
{ "dataset_id", "UUID" + VALUE_CODEC },
{ "producer_id", "UUID" + VALUE_CODEC },
{ "schema_version", "UInt32" + INTEGER_CODEC },
{ "created_at", "DateTime64(3, 'UTC')" + INTEGER_CODEC }
};
private static final String[][] WRITER_REGISTRATION_COLUMN_DEFINITIONS = {
{ "dataset_id", "UUID" },
{ "producer_id", "UUID" },
{ "writer_id", "UUID" },
{ "registration_order", "UInt64 DEFAULT generateSnowflakeID()" },
{ "registered_at", "DateTime64(3, 'UTC')" }
};
private static final String[][] RETENTION_HIGH_WATER_COLUMN_DEFINITIONS = {
{ "dataset_id", "UUID" + VALUE_CODEC },
{ "producer_id", "UUID" + VALUE_CODEC },
{ "producer_sequence", "UInt64" + INTEGER_CODEC },
{ "family", "LowCardinality(String)" + VALUE_CODEC },
{ "rowid", "UInt64" + INTEGER_CODEC },
{ "recorded_at", "DateTime64(3, 'UTC')" + INTEGER_CODEC }
};
private static final String[][] EVENT_COLUMN_DEFINITIONS = {
{ "dataset_id", "UUID" + VALUE_CODEC },
{ "producer_id", "UUID" + VALUE_CODEC },
{ "producer_sequence", "UInt64" + INTEGER_CODEC },
{ "batch_id", "UUID" + VALUE_CODEC },
{ "batch_ordinal", "UInt32" + INTEGER_CODEC },
{ "family", "LowCardinality(String)" + VALUE_CODEC },
{ "rowid", "UInt64" + INTEGER_CODEC },
{ "time", "UInt32" + INTEGER_CODEC },
{ "user_id", "Nullable(UInt32)" + VALUE_CODEC },
{ "wid", "UInt32" + INTEGER_CODEC },
{ "x", "Int32" + INTEGER_CODEC },
{ "y", "Nullable(Int32)" + VALUE_CODEC },
{ "z", "Int32" + INTEGER_CODEC },
{ "type", "Nullable(UInt32)" + VALUE_CODEC },
{ "data", "Nullable(Int64)" + VALUE_CODEC },
{ "payload", "Nullable(String)" + VALUE_CODEC },
{ "meta", "Nullable(String)" + VALUE_CODEC },
{ "blockdata", "Nullable(String)" + VALUE_CODEC },
{ "action", "Nullable(UInt8)" + VALUE_CODEC },
{ "rolled_back", "Nullable(UInt8)" + VALUE_CODEC },
{ "amount", "Nullable(Int32)" + VALUE_CODEC },
{ "metadata", "Nullable(String)" + VALUE_CODEC },
{ "entity_spawn_rowid", "Nullable(UInt64)" + VALUE_CODEC },
{ "id", "Nullable(UInt32)" + VALUE_CODEC },
{ "name", "Nullable(String)" + VALUE_CODEC },
{ "text", "Nullable(String)" + VALUE_CODEC },
{ "message", "Nullable(String)" + VALUE_CODEC },
{ "status", "Nullable(UInt8)" + VALUE_CODEC },
{ "database_lock_time", "Nullable(UInt32)" + VALUE_CODEC },
{ "version", "Nullable(String)" + VALUE_CODEC },
{ "block_rowid", "Nullable(UInt64)" + VALUE_CODEC },
{ "kill_rowid", "Nullable(UInt64)" + VALUE_CODEC },
{ "block_rowid_present", "Nullable(UInt8)" + VALUE_CODEC },
{ "kill_rowid_present", "Nullable(UInt8)" + VALUE_CODEC },
{ "uuid", "Nullable(String)" + VALUE_CODEC },
{ "user_name", "Nullable(String)" + VALUE_CODEC },
{ "current_wid", "Nullable(UInt32)" + VALUE_CODEC },
{ "origin_x", "Nullable(Float64)" + VALUE_CODEC },
{ "origin_y", "Nullable(Float64)" + VALUE_CODEC },
{ "origin_z", "Nullable(Float64)" + VALUE_CODEC },
{ "current_x", "Nullable(Float64)" + VALUE_CODEC },
{ "current_y", "Nullable(Float64)" + VALUE_CODEC },
{ "current_z", "Nullable(Float64)" + VALUE_CODEC },
{ "yaw", "Nullable(Float32)" + VALUE_CODEC },
{ "pitch", "Nullable(Float32)" + VALUE_CODEC },
{ "entity_data", "Nullable(String)" + VALUE_CODEC },
{ "entity_data_present", "Nullable(UInt8)" + VALUE_CODEC },
{ "removed", "Nullable(UInt8)" + VALUE_CODEC },
{ "color", "Nullable(UInt32)" + VALUE_CODEC },
{ "color_secondary", "Nullable(UInt32)" + VALUE_CODEC },
{ "sign_data", "Nullable(UInt8)" + VALUE_CODEC },
{ "waxed", "Nullable(UInt8)" + VALUE_CODEC },
{ "face", "Nullable(UInt8)" + VALUE_CODEC },
{ "line_1", "Nullable(String)" + VALUE_CODEC },
{ "line_2", "Nullable(String)" + VALUE_CODEC },
{ "line_3", "Nullable(String)" + VALUE_CODEC },
{ "line_4", "Nullable(String)" + VALUE_CODEC },
{ "line_5", "Nullable(String)" + VALUE_CODEC },
{ "line_6", "Nullable(String)" + VALUE_CODEC },
{ "line_7", "Nullable(String)" + VALUE_CODEC },
{ "line_8", "Nullable(String)" + VALUE_CODEC }
};
public static final List<String> EVENT_COLUMNS = eventColumns();
public static final List<String> EVENT_COLUMN_TYPES = eventColumnTypes();
static final int PHYSICAL_TABLE_COUNT = 4;
private ClickHouseSchema() {
throw new IllegalStateException("Schema class");
}
public static List<String> createStatements(String database, String prefix) {
Names names = new Names(database, prefix);
List<String> statements = new ArrayList<>();
statements.add(createStorageMetadata(names));
statements.add(createWriterRegistration(names));
statements.add(createRetentionHighWater(names));
statements.add(createEventData(names));
addCompatibilityViews(statements, names);
return Collections.unmodifiableList(statements);
}
static void validatePhysicalSchema(Connection connection, String database, String prefix) throws SQLException {
Names names = new Names(database, prefix);
validateTable(connection, database, names.rawTable("storage_metadata"), "MergeTree", "tuple()", "", STORAGE_METADATA_COLUMN_DEFINITIONS,
"fsync_after_insert=1", "fsync_part_directory=1");
validateTable(connection, database, names.rawTable("writer_registration"), "MergeTree", "(registration_order,writer_id)", "", WRITER_REGISTRATION_COLUMN_DEFINITIONS,
"fsync_after_insert=1", "fsync_part_directory=1");
validateTable(connection, database, names.rawTable("retention_high_water"), "MergeTree", "(dataset_id,family,producer_sequence,rowid)", "", RETENTION_HIGH_WATER_COLUMN_DEFINITIONS,
"fsync_after_insert=1", "fsync_part_directory=1");
validateTable(connection, database, names.rawTable("event_data"), "CoalescingMergeTree", EVENT_SORTING_KEY, EVENT_PARTITION_KEY, EVENT_COLUMN_DEFINITIONS,
"fsync_after_insert=1", "fsync_part_directory=1", "non_replicated_deduplication_window=1000");
validateDataSkippingIndexes(connection, database, names.rawTable("event_data"), EVENT_DATA_SKIPPING_INDEX_DEFINITIONS);
}
static void requireUnownedTablesEmpty(Connection connection, String database, String prefix) throws SQLException {
Names names = new Names(database, prefix);
for (String suffix : new String[] { "writer_registration", "retention_high_water", "event_data" }) {
try (PreparedStatement statement = connection.prepareStatement("SELECT 1 FROM " + names.table(suffix) + " LIMIT 1"); ResultSet resultSet = statement.executeQuery()) {
if (resultSet.next()) {
throw new SQLException("ClickHouse table " + names.rawTable(suffix) + " contains data without CoreProtect storage metadata");
}
}
}
}
private static void validateTable(Connection connection, String database, String table, String expectedEngine, String expectedSortingKey, String expectedPartitionKey,
String[][] expectedColumns, String... requiredSettings) throws SQLException {
String tableQuery = "SELECT engine,sorting_key,partition_key,create_table_query FROM system.tables WHERE database=? AND name=? LIMIT 2";
try (PreparedStatement statement = connection.prepareStatement(tableQuery)) {
statement.setString(1, database);
statement.setString(2, table);
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse schema table is missing: " + table);
}
String engine = resultSet.getString(1);
String sortingKey = resultSet.getString(2);
String partitionKey = resultSet.getString(3);
String createQuery = normalizeSql(resultSet.getString(4));
if (resultSet.next()) {
throw new SQLException("ClickHouse schema table is ambiguous: " + table);
}
if (!expectedEngine.equals(engine)
|| !normalizeKey(expectedSortingKey).equals(normalizeKey(sortingKey))
|| !normalizeExpression(expectedPartitionKey).equals(normalizeExpression(partitionKey))) {
throw new SQLException("ClickHouse table has an incompatible engine or key definition: " + table);
}
for (String setting : requiredSettings) {
if (!createQuery.contains(normalizeSql(setting))) {
throw new SQLException("ClickHouse table is missing required setting " + setting + ": " + table);
}
}
}
}
String columnQuery = "SELECT name,type,default_kind,default_expression FROM system.columns WHERE database=? AND table=? ORDER BY position";
try (PreparedStatement statement = connection.prepareStatement(columnQuery)) {
statement.setString(1, database);
statement.setString(2, table);
try (ResultSet resultSet = statement.executeQuery()) {
for (String[] expectedColumn : expectedColumns) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse table has missing columns: " + table);
}
String definition = expectedColumn[1];
String expectedDefault = defaultExpression(definition);
String defaultKind = resultSet.getString(3);
String defaultExpression = resultSet.getString(4);
boolean defaultMatches = expectedDefault == null
? defaultKind == null || defaultKind.isEmpty()
: "DEFAULT".equalsIgnoreCase(defaultKind) && normalizeSql(expectedDefault).equals(normalizeSql(defaultExpression));
if (!expectedColumn[0].equals(resultSet.getString(1))
|| !normalizeSql(columnType(definition)).equals(normalizeSql(resultSet.getString(2)))
|| !defaultMatches) {
throw new SQLException("ClickHouse table has an incompatible column definition: " + table + "." + expectedColumn[0]);
}
}
if (resultSet.next()) {
throw new SQLException("ClickHouse table has unexpected columns: " + table);
}
}
}
}
private static String createStorageMetadata(Names names) {
return table(names.storageMetadata, columnDefinitions(STORAGE_METADATA_COLUMN_DEFINITIONS))
+ " ENGINE = MergeTree"
+ " ORDER BY tuple()"
+ " SETTINGS fsync_after_insert=1,fsync_part_directory=1";
}
private static String createWriterRegistration(Names names) {
return table(names.writerRegistration, columnDefinitions(WRITER_REGISTRATION_COLUMN_DEFINITIONS))
+ " ENGINE = MergeTree"
+ " ORDER BY (registration_order,writer_id)"
+ " SETTINGS fsync_after_insert=1,fsync_part_directory=1";
}
private static String createRetentionHighWater(Names names) {
return table(names.retentionHighWater, columnDefinitions(RETENTION_HIGH_WATER_COLUMN_DEFINITIONS))
+ " ENGINE = MergeTree"
+ " ORDER BY (dataset_id,family,producer_sequence,rowid)"
+ " SETTINGS fsync_after_insert=1,fsync_part_directory=1";
}
private static String createEventData(Names names) {
String[] columns = eventColumnDefinitions();
String[] tableElements = new String[columns.length + EVENT_DATA_SKIPPING_INDEX_DEFINITIONS.length];
System.arraycopy(columns, 0, tableElements, 0, columns.length);
for (int index = 0; index < EVENT_DATA_SKIPPING_INDEX_DEFINITIONS.length; index++) {
String[] definition = EVENT_DATA_SKIPPING_INDEX_DEFINITIONS[index];
tableElements[columns.length + index] = "INDEX " + definition[0] + " " + definition[1] + " TYPE " + definition[2] + " GRANULARITY " + definition[3];
}
return table(names.eventData, tableElements)
+ " ENGINE = CoalescingMergeTree"
+ " PARTITION BY " + EVENT_PARTITION_KEY
+ " ORDER BY " + EVENT_SORTING_KEY
+ " SETTINGS fsync_after_insert=1,fsync_part_directory=1,non_replicated_deduplication_window=1000";
}
private static void validateDataSkippingIndexes(Connection connection, String database, String table, String[][] expectedIndexes) throws SQLException {
String indexQuery = "SELECT type_full,expr,granularity FROM system.data_skipping_indices WHERE database=? AND table=? AND name=? LIMIT 2";
try (PreparedStatement statement = connection.prepareStatement(indexQuery)) {
for (String[] expectedIndex : expectedIndexes) {
statement.setString(1, database);
statement.setString(2, table);
statement.setString(3, expectedIndex[0]);
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse table is missing required data-skipping index " + expectedIndex[0] + ": " + table);
}
boolean matches = normalizeSql(expectedIndex[2]).equals(normalizeSql(resultSet.getString(1)))
&& normalizeExpression(expectedIndex[1]).equals(normalizeExpression(resultSet.getString(2)))
&& Long.parseLong(expectedIndex[3]) == resultSet.getLong(3);
if (!matches || resultSet.next()) {
throw new SQLException("ClickHouse table has an incompatible data-skipping index " + expectedIndex[0] + ": " + table);
}
}
}
}
}
private static void addCompatibilityViews(List<String> statements, Names names) {
statements.add(view(names, ClickHouseFamily.ART_MAP, "e.rowid AS rowid,e.id AS id,e.name AS art"));
statements.add(rollbackView(names, ClickHouseFamily.BLOCK, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.type AS type,e.data AS data," + binary("e.meta", "meta") + "," + binary("e.blockdata", "blockdata") + ",e.action AS action"));
statements.add(view(names, ClickHouseFamily.CHAT, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.message AS message"));
statements.add(view(names, ClickHouseFamily.COMMAND, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.message AS message"));
statements.add(rollbackView(names, ClickHouseFamily.CONTAINER, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.type AS type,e.data AS data,e.amount AS amount," + binary("e.metadata", "metadata") + ",e.action AS action"));
statements.add(rollbackView(names, ClickHouseFamily.ENTITY_CONTAINER, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.entity_spawn_rowid AS entity_spawn_rowid,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.type AS type,e.data AS data,e.amount AS amount," + binary("e.metadata", "metadata") + ",e.action AS action"));
statements.add(view(names, ClickHouseFamily.ENTITY_INTERACTION, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.entity_spawn_rowid AS entity_spawn_rowid,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.type AS type,e.action AS action," + binary("e.metadata", "metadata") + ",e.rolled_back AS rolled_back"));
statements.add(rollbackView(names, ClickHouseFamily.ITEM, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.type AS type," + binary("e.payload", "data") + ",e.amount AS amount,e.action AS action"));
statements.add(currentView(names, ClickHouseFamily.DATABASE_LOCK, "e.rowid AS rowid,e.status AS status,e.database_lock_time AS time"));
statements.add(view(names, ClickHouseFamily.ENTITY, "e.rowid AS rowid,e.time AS time," + binary("e.payload", "data")));
statements.add(entitySpawnView(names));
statements.add(view(names, ClickHouseFamily.ENTITY_MAP, "e.rowid AS rowid,e.id AS id,e.name AS entity"));
statements.add(view(names, ClickHouseFamily.MATERIAL_MAP, "e.rowid AS rowid,e.id AS id,e.name AS material"));
statements.add(view(names, ClickHouseFamily.BLOCKDATA_MAP, "e.rowid AS rowid,e.id AS id,e.text AS data"));
statements.add(view(names, ClickHouseFamily.SESSION, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.action AS action"));
statements.add(view(names, ClickHouseFamily.SIGN, "e.rowid AS rowid,e.time AS time,e.user_id AS `user`,e.wid AS wid,e.x AS x,e.y AS y,e.z AS z,e.action AS action,e.color AS color,e.color_secondary AS color_secondary,e.sign_data AS data,e.waxed AS waxed,e.face AS face,e.line_1 AS line_1,e.line_2 AS line_2,e.line_3 AS line_3,e.line_4 AS line_4,e.line_5 AS line_5,e.line_6 AS line_6,e.line_7 AS line_7,e.line_8 AS line_8"));
statements.add(view(names, ClickHouseFamily.SKULL, "e.rowid AS rowid,e.time AS time,e.name AS owner,e.text AS skin"));
statements.add(currentView(names, ClickHouseFamily.USER, "e.rowid AS rowid,e.time AS time,e.user_name AS `user`,e.uuid AS uuid"));
statements.add(view(names, ClickHouseFamily.USERNAME_LOG, "e.rowid AS rowid,e.time AS time,e.uuid AS uuid,e.user_name AS `user`"));
statements.add(currentView(names, ClickHouseFamily.VERSION, "e.rowid AS rowid,e.time AS time,e.version AS version"));
statements.add(view(names, ClickHouseFamily.WORLD, "e.rowid AS rowid,e.id AS id,e.name AS world"));
}
private static String view(Names names, ClickHouseFamily family, String projection) {
return "CREATE OR REPLACE VIEW " + names.table(family.getTableName())
+ " AS SELECT " + projection
+ " FROM " + events(names, family) + " AS e";
}
private static String currentView(Names names, ClickHouseFamily family, String projection) {
return "CREATE OR REPLACE VIEW " + names.table(family.getTableName())
+ " AS SELECT " + projection
+ " FROM " + currentEvents(names, family) + " AS e";
}
private static String rollbackView(Names names, ClickHouseFamily family, String projection) {
return currentView(names, family, projection + ",e.rolled_back AS rolled_back");
}
private static String entitySpawnView(Names names) {
return "CREATE OR REPLACE VIEW " + names.table(ClickHouseFamily.ENTITY_SPAWN.getTableName())
+ " AS SELECT e.rowid AS rowid,e.time AS time"
+ ",if(e.block_rowid_present=1,e.block_rowid,NULL) AS block_rowid"
+ ",if(e.kill_rowid_present=1,e.kill_rowid,NULL) AS kill_rowid"
+ ",e.uuid AS uuid,e.wid AS wid,e.current_wid AS current_wid"
+ ",e.origin_x AS origin_x,e.origin_y AS origin_y,e.origin_z AS origin_z"
+ ",e.current_x AS x,e.current_y AS y,e.current_z AS z"
+ ",e.yaw AS yaw,e.pitch AS pitch," + binary("if(e.entity_data_present=1,e.entity_data,NULL)", "data") + ",e.removed AS removed"
+ " FROM " + currentEvents(names, ClickHouseFamily.ENTITY_SPAWN) + " AS e";
}
static String binary(String value, String alias) {
String presentValue = "ifNull(" + value + ",'')";
String bytes = "arrayMap(i -> reinterpretAsInt8(substring(" + presentValue + ",i,1)),range(1,length(" + presentValue + ")+1))";
return "if(isNull(" + value + "),CAST([], 'Array(Int8)'),arrayConcat([toInt8(0)]," + bytes + ")) AS " + alias;
}
private static String events(Names names, ClickHouseFamily family) {
return "(SELECT * FROM " + names.eventData + " WHERE family='" + family.getTableName() + "')";
}
private static String currentEvents(Names names, ClickHouseFamily family) {
return "(SELECT * FROM " + names.eventData + " FINAL WHERE family='" + family.getTableName() + "')";
}
private static String table(String name, String... columns) {
return "CREATE TABLE IF NOT EXISTS " + name + " (" + String.join(",", columns) + ")";
}
private static List<String> eventColumns() {
List<String> columns = new ArrayList<>(EVENT_COLUMN_DEFINITIONS.length);
for (String[] definition : EVENT_COLUMN_DEFINITIONS) {
columns.add(definition[0]);
}
return Collections.unmodifiableList(columns);
}
private static List<String> eventColumnTypes() {
return columnTypes(EVENT_COLUMN_DEFINITIONS);
}
private static String[] eventColumnDefinitions() {
return columnDefinitions(EVENT_COLUMN_DEFINITIONS);
}
private static List<String> columnTypes(String[][] columns) {
List<String> types = new ArrayList<>(columns.length);
for (String[] column : columns) {
types.add(columnType(column[1]));
}
return Collections.unmodifiableList(types);
}
private static String columnType(String definition) {
int end = definition.length();
int codec = definition.indexOf(" CODEC(");
int defaultValue = definition.indexOf(" DEFAULT ");
if (codec >= 0) {
end = Math.min(end, codec);
}
if (defaultValue >= 0) {
end = Math.min(end, defaultValue);
}
return definition.substring(0, end);
}
private static String defaultExpression(String definition) {
int start = definition.indexOf(" DEFAULT ");
return start < 0 ? null : definition.substring(start + " DEFAULT ".length());
}
private static String[] columnDefinitions(String[][] columns) {
String[] definitions = new String[columns.length];
for (int index = 0; index < columns.length; index++) {
definitions[index] = columns[index][0] + " " + columns[index][1];
}
return definitions;
}
private static String purgeableFamilies() {
StringJoiner families = new StringJoiner(",");
for (ClickHouseFamily family : ClickHouseFamily.values()) {
if (family.isPurgeable()) {
families.add("'" + family.getTableName() + "'");
}
}
return families.toString();
}
private static String normalizeKey(String value) {
String normalized = normalizeExpression(value);
if (normalized.startsWith("tuple(") && normalized.endsWith(")")) {
normalized = normalized.substring("tuple".length());
}
if (normalized.startsWith("(") && normalized.endsWith(")")) {
normalized = normalized.substring(1, normalized.length() - 1);
}
return normalized;
}
private static String normalizeExpression(String value) {
String normalized = normalizeSql(value);
int offset = 0;
while ((offset = normalized.indexOf("if((", offset)) >= 0) {
int open = offset + 3;
int depth = 0;
int close = -1;
for (int index = open; index < normalized.length(); index++) {
char character = normalized.charAt(index);
if (character == '(') {
depth++;
}
else if (character == ')' && --depth == 0) {
close = index;
break;
}
}
if (close < 0 || close + 1 >= normalized.length() || normalized.charAt(close + 1) != ',') {
offset = open + 1;
continue;
}
normalized = normalized.substring(0, open) + normalized.substring(open + 1, close) + normalized.substring(close + 1);
}
return normalized;
}
private static String normalizeSql(String value) {
if (value == null || value.isEmpty()) {
return "";
}
StringBuilder normalized = new StringBuilder(value.length());
for (int index = 0; index < value.length(); index++) {
char character = value.charAt(index);
if (!Character.isWhitespace(character) && character != '`') {
normalized.append(character);
}
}
return normalized.toString().toLowerCase(Locale.ROOT);
}
private static final class Names {
private final String database;
private final String prefix;
private final String eventData;
private final String retentionHighWater;
private final String storageMetadata;
private final String writerRegistration;
private Names(String database, String prefix) {
this.database = ClickHouseIdentifiers.quote(database, "ClickHouse database");
this.prefix = validatePrefix(prefix);
eventData = table("event_data");
retentionHighWater = table("retention_high_water");
storageMetadata = table("storage_metadata");
writerRegistration = table("writer_registration");
}
private String table(String suffix) {
return database + "." + ClickHouseIdentifiers.quote(rawTable(suffix), "ClickHouse table");
}
private String rawTable(String suffix) {
return prefix + suffix;
}
private static String validatePrefix(String prefix) {
if (prefix == null) {
throw new IllegalArgumentException("ClickHouse table prefix cannot be null");
}
return prefix.isEmpty() ? prefix : ClickHouseIdentifiers.requireIdentifier(prefix, "ClickHouse table prefix");
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseStartupReconciler.java
+109
View File
@@ -0,0 +1,109 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.EnumMap;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;
public final class ClickHouseStartupReconciler {
private ClickHouseStartupReconciler() {
throw new IllegalStateException("Utility class");
}
public static ClickHouseHighWaterMarks readRemote(Connection connection, String database, String prefix, UUID datasetId, UUID producerId) throws SQLException {
Objects.requireNonNull(connection, "connection");
Objects.requireNonNull(datasetId, "datasetId");
Objects.requireNonNull(producerId, "producerId");
String validatedPrefix = prefix == null || prefix.isEmpty() ? "" : ClickHouseIdentifiers.requireIdentifier(prefix, "ClickHouse table prefix");
String eventTable = table(database, validatedPrefix, "event_data");
String retentionHighWater = table(database, validatedPrefix, "retention_high_water");
rejectDifferentDataset(connection, eventTable, datasetId);
rejectDifferentProducer(connection, eventTable, datasetId, producerId);
return new ClickHouseHighWaterMarks(readProducerSequence(connection, eventTable, retentionHighWater, datasetId, producerId), readRawRowIds(connection, eventTable, retentionHighWater, datasetId));
}
private static void rejectDifferentDataset(Connection connection, String table, UUID datasetId) throws SQLException {
String sql = "SELECT dataset_id FROM " + table + " WHERE dataset_id != ? LIMIT 1";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
try (ResultSet resultSet = statement.executeQuery()) {
if (resultSet.next()) {
throw new SQLException("ClickHouse table namespace is already assigned to dataset " + resultSet.getString(1));
}
}
}
}
private static void rejectDifferentProducer(Connection connection, String table, UUID datasetId, UUID producerId) throws SQLException {
String sql = "SELECT producer_id FROM " + table + " WHERE dataset_id=? AND producer_id != ? LIMIT 1";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.setObject(2, producerId);
try (ResultSet resultSet = statement.executeQuery()) {
if (resultSet.next()) {
throw new SQLException("ClickHouse dataset " + datasetId + " is already written by producer " + resultSet.getString(1));
}
}
}
}
private static long readProducerSequence(Connection connection, String eventTable, String retentionHighWater, UUID datasetId, UUID producerId) throws SQLException {
String sql = "SELECT max(producer_sequence) FROM ("
+ "SELECT producer_sequence FROM " + eventTable + " WHERE dataset_id=? AND producer_id=? UNION ALL "
+ "SELECT producer_sequence FROM " + retentionHighWater + " WHERE dataset_id=? AND producer_id=?)";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.setObject(2, producerId);
statement.setObject(3, datasetId);
statement.setObject(4, producerId);
try (ResultSet resultSet = statement.executeQuery()) {
return resultSet.next() ? requireNonNegative(resultSet.getLong(1), "producer sequence") : 0;
}
}
}
private static Map<ClickHouseFamily, Long> readRawRowIds(Connection connection, String eventTable, String retentionHighWater, UUID datasetId) throws SQLException {
String sql = "SELECT family,max(rowid) FROM ("
+ "SELECT family,rowid FROM " + eventTable + " WHERE dataset_id=? UNION ALL "
+ "SELECT family,rowid FROM " + retentionHighWater + " WHERE dataset_id=?) GROUP BY family";
Map<ClickHouseFamily, Long> rowIds = new EnumMap<>(ClickHouseFamily.class);
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.setObject(2, datasetId);
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
putMax(rowIds, resultSet.getString(1), resultSet.getLong(2));
}
}
}
return rowIds;
}
private static void putMax(Map<ClickHouseFamily, Long> rowIds, String familyName, long rowId) throws SQLException {
ClickHouseFamily family;
try {
family = ClickHouseFamily.fromTableName(familyName);
}
catch (IllegalArgumentException exception) {
throw new SQLException("Unknown ClickHouse event family in identity storage", exception);
}
rowIds.merge(family, requireNonNegative(rowId, "compatibility row ID"), Math::max);
}
private static long requireNonNegative(long value, String name) throws SQLException {
if (value < 0) {
throw new SQLException("ClickHouse " + name + " exceeds the supported signed 64-bit range");
}
return value;
}
private static String table(String database, String prefix, String suffix) {
return ClickHouseIdentifiers.qualified(database, prefix + suffix);
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseStateBatch.java
+240
View File
@@ -0,0 +1,240 @@
package net.coreprotect.database.clickhouse;
import java.sql.SQLException;
import java.util.EnumMap;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;
final class ClickHouseStateBatch implements AutoCloseable {
private final ClickHouseBatchIdentity identity;
private final EnumMap<ClickHouseFamily, LinkedHashMap<Long, RollbackUpdate>> rollbackUpdates = new EnumMap<>(ClickHouseFamily.class);
private final LinkedHashMap<Long, ClickHouseEntityState> entityStateUpdates = new LinkedHashMap<>();
private boolean sealed;
ClickHouseStateBatch(ClickHouseBatchIdentity identity) {
this.identity = Objects.requireNonNull(identity, "identity");
}
void addRollback(ClickHouseEventPointer pointer, int rolledBack) {
ensureWritable();
Objects.requireNonNull(pointer, "pointer");
if (!identity.getDatasetId().equals(pointer.getDatasetId())) {
throw new IllegalArgumentException("ClickHouse rollback target belongs to a different dataset");
}
addRollback(new RollbackUpdate(pointer, rolledBack));
}
void addRollback(ClickHouseFamily family, long rowId, int time, int worldId, int x, int z, int rolledBack) {
ensureWritable();
addRollback(new RollbackUpdate(family, rowId, time, worldId, x, z, rolledBack));
}
void addEntityState(ClickHouseEntityState state) {
ensureWritable();
Objects.requireNonNull(state, "state");
ClickHouseEventPointer pointer = state.getPointer();
if (!identity.getDatasetId().equals(pointer.getDatasetId())) {
throw new IllegalArgumentException("ClickHouse entity state belongs to a different dataset");
}
if (pointer.getFamily() != ClickHouseFamily.ENTITY_SPAWN) {
throw new IllegalArgumentException("ClickHouse entity state requires an entity spawn target");
}
entityStateUpdates.put(pointer.getRowId(), state);
}
void appendTo(ClickHouseRowBinaryBuffer rows, int firstOrdinal) throws SQLException {
if (sealed) {
throw new IllegalStateException("ClickHouse state batch is already appended");
}
ensureWritable();
Objects.requireNonNull(rows, "rows");
int ordinal = firstOrdinal;
for (Map<Long, RollbackUpdate> familyUpdates : rollbackUpdates.values()) {
for (RollbackUpdate update : familyUpdates.values()) {
beginSparseRow(rows, update, ordinal++);
rows.set("rolled_back", update.rolledBack);
rows.commitRow("rollback state update");
}
}
for (ClickHouseEntityState state : entityStateUpdates.values()) {
ClickHouseEventPointer pointer = state.getPointer();
beginSparseRow(rows, pointer, ordinal++);
Long blockRowId = state.getBlockRowId();
Integer killRowId = state.getKillRowId();
rows.set("block_rowid", blockRowId);
rows.set("kill_rowid", killRowId == null ? null : killRowId.longValue());
rows.set("block_rowid_present", blockRowId == null ? 0 : 1);
rows.set("kill_rowid_present", killRowId == null ? 0 : 1);
rows.set("uuid", state.getUuid().toString());
rows.set("current_wid", state.getWorldId());
rows.set("current_x", state.getX());
rows.set("current_y", state.getY());
rows.set("current_z", state.getZ());
rows.set("yaw", state.getYaw());
rows.set("pitch", state.getPitch());
byte[] data = state.getData();
rows.set("entity_data", data);
rows.set("entity_data_present", data == null ? 0 : 1);
rows.set("removed", state.isRemoved() ? 1 : 0);
rows.commitRow("entity state update");
}
sealed = true;
}
int getRollbackCount() {
int count = 0;
for (Map<Long, RollbackUpdate> updates : rollbackUpdates.values()) {
count += updates.size();
}
return count;
}
int getEntityStateCount() {
return entityStateUpdates.size();
}
int getLocalOverlapCount(int eventCount) {
int count = 0;
for (Map<Long, RollbackUpdate> updates : rollbackUpdates.values()) {
for (RollbackUpdate update : updates.values()) {
if (isLocal(update, eventCount)) {
count++;
}
}
}
for (ClickHouseEntityState state : entityStateUpdates.values()) {
if (isLocal(state.getPointer(), eventCount)) {
count++;
}
}
return count;
}
Checkpoint checkpoint() {
ensureWritable();
return new Checkpoint(rollbackUpdates, entityStateUpdates);
}
void restore(Checkpoint checkpoint) {
ensureWritable();
Objects.requireNonNull(checkpoint, "checkpoint");
rollbackUpdates.clear();
checkpoint.rollbackUpdates.forEach((family, updates) -> rollbackUpdates.put(family, new LinkedHashMap<>(updates)));
entityStateUpdates.clear();
entityStateUpdates.putAll(checkpoint.entityStateUpdates);
}
@Override
public void close() {
sealed = true;
rollbackUpdates.clear();
entityStateUpdates.clear();
}
private void addRollback(RollbackUpdate update) {
if (!isRollbackFamily(update.family)) {
throw new IllegalArgumentException("ClickHouse rollback state is unsupported for " + update.family.getTableName());
}
if (update.rowId < 1) {
throw new IllegalArgumentException("ClickHouse rollback row IDs must be positive");
}
if (update.rolledBack < 0 || update.rolledBack > 3) {
throw new IllegalArgumentException("CoreProtect rolled-back state must be between 0 and 3");
}
rollbackUpdates.computeIfAbsent(update.family, ignored -> new LinkedHashMap<>()).put(update.rowId, update);
}
private void beginSparseRow(ClickHouseRowBinaryBuffer rows, RollbackUpdate update, int ordinal) {
beginSparseRow(rows, update.family, update.rowId, update.time, update.worldId, update.x, update.z, ordinal);
}
private void beginSparseRow(ClickHouseRowBinaryBuffer rows, ClickHouseEventPointer pointer, int ordinal) {
beginSparseRow(rows, pointer.getFamily(), pointer.getRowId(), pointer.getTime(), pointer.getWorldId(), pointer.getX(), pointer.getZ(), ordinal);
}
private void beginSparseRow(ClickHouseRowBinaryBuffer rows, ClickHouseFamily family, long rowId, int time, int worldId, int x, int z, int ordinal) {
rows.beginRow();
rows.set("dataset_id", identity.getDatasetId());
rows.set("producer_id", identity.getProducerId());
rows.set("producer_sequence", identity.getProducerSequence());
rows.set("batch_id", identity.getBatchId());
rows.set("batch_ordinal", ordinal);
rows.set("family", family.getTableName());
rows.set("rowid", rowId);
rows.set("time", time);
rows.set("wid", worldId);
rows.set("x", x);
rows.set("z", z);
}
private void ensureWritable() {
if (sealed) {
throw new IllegalStateException("ClickHouse state batch is sealed");
}
}
private boolean isLocal(RollbackUpdate update, int eventCount) {
return update.pointer != null && isLocal(update.pointer, eventCount);
}
private boolean isLocal(ClickHouseEventPointer pointer, int eventCount) {
return identity.getProducerId().equals(pointer.getProducerId())
&& identity.getProducerSequence() == pointer.getProducerSequence()
&& pointer.getBatchOrdinal() < eventCount;
}
private static boolean isRollbackFamily(ClickHouseFamily family) {
return family == ClickHouseFamily.BLOCK
|| family == ClickHouseFamily.CONTAINER
|| family == ClickHouseFamily.ENTITY_CONTAINER
|| family == ClickHouseFamily.ITEM;
}
static final class Checkpoint {
private final EnumMap<ClickHouseFamily, LinkedHashMap<Long, RollbackUpdate>> rollbackUpdates = new EnumMap<>(ClickHouseFamily.class);
private final LinkedHashMap<Long, ClickHouseEntityState> entityStateUpdates;
private Checkpoint(EnumMap<ClickHouseFamily, LinkedHashMap<Long, RollbackUpdate>> rollbackUpdates, LinkedHashMap<Long, ClickHouseEntityState> entityStateUpdates) {
rollbackUpdates.forEach((family, updates) -> this.rollbackUpdates.put(family, new LinkedHashMap<>(updates)));
this.entityStateUpdates = new LinkedHashMap<>(entityStateUpdates);
}
}
private static final class RollbackUpdate {
private final ClickHouseFamily family;
private final long rowId;
private final int time;
private final int worldId;
private final int x;
private final int z;
private final ClickHouseEventPointer pointer;
private final int rolledBack;
private RollbackUpdate(ClickHouseEventPointer pointer, int rolledBack) {
this.family = pointer.getFamily();
this.rowId = pointer.getRowId();
this.time = pointer.getTime();
this.worldId = pointer.getWorldId();
this.x = pointer.getX();
this.z = pointer.getZ();
this.pointer = pointer;
this.rolledBack = rolledBack;
}
private RollbackUpdate(ClickHouseFamily family, long rowId, int time, int worldId, int x, int z, int rolledBack) {
this.family = Objects.requireNonNull(family, "family");
this.rowId = rowId;
this.time = time;
this.worldId = worldId;
this.x = x;
this.z = z;
this.pointer = null;
this.rolledBack = rolledBack;
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseStorageIdentity.java
+100
View File
@@ -0,0 +1,100 @@
package net.coreprotect.database.clickhouse;
import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.UUID;
final class ClickHouseStorageIdentity {
private final UUID datasetId;
private final UUID producerId;
private ClickHouseStorageIdentity(UUID datasetId, UUID producerId) {
this.datasetId = datasetId;
this.producerId = producerId;
}
static ClickHouseStorageIdentity loadOrCreate(Connection connection, String database, String prefix) throws SQLException {
String tableName = prefix + "storage_metadata";
String table = ClickHouseIdentifiers.qualified(database, tableName);
ClickHouseStorageIdentity identity = read(connection, table);
if (identity == null) {
UUID datasetId = readTableUuid(connection, database, tableName);
UUID producerId = UUID.nameUUIDFromBytes(("coreprotect-clickhouse-producer-v1:" + datasetId).getBytes(StandardCharsets.UTF_8));
insert(connection, table, datasetId, producerId);
identity = read(connection, table);
}
if (identity == null) {
throw new SQLException("ClickHouse storage identity is not visible after creation");
}
return identity;
}
static ClickHouseStorageIdentity load(Connection connection, String database, String prefix) throws SQLException {
return read(connection, ClickHouseIdentifiers.qualified(database, prefix + "storage_metadata"));
}
UUID getDatasetId() {
return datasetId;
}
UUID getProducerId() {
return producerId;
}
private static ClickHouseStorageIdentity read(Connection connection, String table) throws SQLException {
String sql = "SELECT dataset_id,producer_id,schema_version FROM " + table
+ " GROUP BY dataset_id,producer_id,schema_version LIMIT 2";
try (PreparedStatement statement = connection.prepareStatement(sql); ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
return null;
}
UUID datasetId = UUID.fromString(resultSet.getString(1));
UUID producerId = UUID.fromString(resultSet.getString(2));
int schemaVersion = resultSet.getInt(3);
if (resultSet.next()) {
throw new SQLException("ClickHouse storage metadata contains conflicting identities or schema versions");
}
if (schemaVersion != ClickHouseSchema.VERSION) {
throw new SQLException("Unsupported ClickHouse schema version " + schemaVersion + " (expected " + ClickHouseSchema.VERSION + ")");
}
return new ClickHouseStorageIdentity(datasetId, producerId);
}
}
private static UUID readTableUuid(Connection connection, String database, String tableName) throws SQLException {
String sql = "SELECT uuid FROM system.tables WHERE database=? AND name=? LIMIT 2";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setString(1, database);
statement.setString(2, tableName);
try (ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
throw new SQLException("ClickHouse storage metadata table identity is not available");
}
UUID tableUuid = UUID.fromString(resultSet.getString(1));
if (resultSet.next()) {
throw new SQLException("ClickHouse storage metadata table identity is ambiguous");
}
if (tableUuid.getMostSignificantBits() == 0L && tableUuid.getLeastSignificantBits() == 0L) {
throw new SQLException("ClickHouse storage metadata requires an Atomic database table UUID");
}
return tableUuid;
}
}
}
private static void insert(Connection connection, String table, UUID datasetId, UUID producerId) throws SQLException {
String sql = "INSERT INTO " + table
+ " (dataset_id,producer_id,schema_version,created_at) VALUES (?,?,?,now64(3, 'UTC'))";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.setObject(2, producerId);
statement.setInt(3, ClickHouseSchema.VERSION);
statement.execute();
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseTargetResolver.java
+110
View File
@@ -0,0 +1,110 @@
package net.coreprotect.database.clickhouse;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.StringJoiner;
import java.util.UUID;
final class ClickHouseTargetResolver {
private static final int SELECT_BATCH_SIZE = 5_000;
private final ClickHouseJdbc jdbc;
private final String eventTable;
private final UUID datasetId;
ClickHouseTargetResolver(ClickHouseJdbc jdbc, String database, String prefix, UUID datasetId) {
this.jdbc = Objects.requireNonNull(jdbc, "jdbc");
String validatedPrefix = prefix == null || prefix.isEmpty() ? "" : ClickHouseIdentifiers.requireIdentifier(prefix, "ClickHouse table prefix");
eventTable = ClickHouseIdentifiers.qualified(database, validatedPrefix + "event_data");
this.datasetId = Objects.requireNonNull(datasetId, "datasetId");
}
Map<Long, ClickHouseEventPointer> resolve(ClickHouseFamily family, List<Long> requestedRowIds) throws SQLException {
Objects.requireNonNull(family, "family");
Set<Long> unique = uniqueRowIds(requestedRowIds, "rollback");
List<Long> rowIds = new ArrayList<>(unique);
Map<Long, ClickHouseEventPointer> pointers = new LinkedHashMap<>();
try (Connection connection = jdbc.openConnection()) {
for (int offset = 0; offset < rowIds.size(); offset += SELECT_BATCH_SIZE) {
resolveBatch(connection, family, rowIds.subList(offset, Math.min(offset + SELECT_BATCH_SIZE, rowIds.size())), pointers, null);
}
}
if (pointers.size() != rowIds.size()) {
List<Long> missing = new ArrayList<>(unique);
missing.removeAll(pointers.keySet());
throw new SQLException("Unable to resolve " + missing.size() + " ClickHouse " + family.getTableName() + " row IDs");
}
return pointers;
}
Map<Long, ClickHouseEventPointer> resolveAvailableBlocks(List<Long> requestedRowIds, Map<Long, Integer> actions) throws SQLException {
Objects.requireNonNull(actions, "actions");
Set<Long> unique = uniqueRowIds(requestedRowIds, "block");
List<Long> rowIds = new ArrayList<>(unique);
Map<Long, ClickHouseEventPointer> pointers = new LinkedHashMap<>();
try (Connection connection = jdbc.openConnection()) {
for (int offset = 0; offset < rowIds.size(); offset += SELECT_BATCH_SIZE) {
resolveBatch(connection, ClickHouseFamily.BLOCK, rowIds.subList(offset, Math.min(offset + SELECT_BATCH_SIZE, rowIds.size())), pointers, actions);
}
}
return pointers;
}
private void resolveBatch(Connection connection, ClickHouseFamily family, List<Long> rowIds, Map<Long, ClickHouseEventPointer> pointers, Map<Long, Integer> actions) throws SQLException {
if (rowIds.isEmpty()) {
return;
}
StringJoiner placeholders = new StringJoiner(",");
for (int ignored = 0; ignored < rowIds.size(); ignored++) {
placeholders.add("?");
}
String sql = "SELECT rowid,producer_id,producer_sequence,batch_ordinal,time,wid,x,z" + (actions == null ? "" : ",action") + " FROM " + eventTable + " FINAL"
+ " WHERE dataset_id=? AND family=?"
+ " AND rowid IN(" + placeholders + ")";
try (PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.setString(2, family.getTableName());
for (int index = 0; index < rowIds.size(); index++) {
statement.setLong(index + 3, rowIds.get(index));
}
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
long rowId = resultSet.getLong(1);
ClickHouseEventPointer pointer = new ClickHouseEventPointer(datasetId, family, UUID.fromString(resultSet.getString(2)), resultSet.getLong(3), resultSet.getInt(4), rowId, resultSet.getInt(5), resultSet.getInt(6), resultSet.getInt(7), resultSet.getInt(8));
if (pointers.putIfAbsent(rowId, pointer) != null) {
throw new SQLException("ClickHouse " + family.getTableName() + " row ID " + rowId + " resolves to multiple committed facts");
}
if (actions != null) {
Object action = resultSet.getObject(9);
if (!(action instanceof Number)) {
throw new SQLException("ClickHouse block row " + rowId + " has no action");
}
actions.put(rowId, ((Number) action).intValue());
}
}
}
}
}
private static Set<Long> uniqueRowIds(List<Long> requestedRowIds, String target) {
Set<Long> unique = new LinkedHashSet<>();
for (Long rowId : Objects.requireNonNull(requestedRowIds, "requestedRowIds")) {
if (rowId == null || rowId < 1) {
throw new IllegalArgumentException("ClickHouse " + target + " targets require positive row IDs");
}
unique.add(rowId);
}
return unique;
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseWriteBatch.java
+127
View File
@@ -0,0 +1,127 @@
package net.coreprotect.database.clickhouse;
import java.io.InputStream;
import java.sql.SQLException;
import java.util.Objects;
public final class ClickHouseWriteBatch implements AutoCloseable {
private final ClickHouseBatchIdentity identity;
private final ClickHouseEventBatch events;
private final ClickHouseStateBatch state;
private ClickHouseBatchReceipt receipt;
private boolean closed;
private boolean published;
ClickHouseWriteBatch(ClickHouseBatchIdentity identity, ClickHouseRowIdAllocator rowIdAllocator) {
this.identity = Objects.requireNonNull(identity, "identity");
events = new ClickHouseEventBatch(identity, rowIdAllocator);
state = new ClickHouseStateBatch(identity);
}
public ClickHouseEventBatch events() {
ensureOpen();
return events;
}
public void addRollback(ClickHouseEventPointer pointer, int rolledBack) throws SQLException {
ensureOpen();
state.addRollback(pointer, rolledBack);
}
void addRollback(ClickHouseFamily family, long rowId, int time, int worldId, int x, int z, int rolledBack) throws SQLException {
ensureOpen();
state.addRollback(family, rowId, time, worldId, x, z, rolledBack);
}
public void addEntityState(ClickHouseEntityState entityState) throws SQLException {
ensureOpen();
state.addEntityState(entityState);
}
ClickHouseBatchReceipt seal() throws SQLException {
ensureOpen();
if (receipt != null) {
return receipt;
}
int eventCount = events.size();
events.seal(state);
int rollbackCount = state.getRollbackCount();
int entityStateCount = state.getEntityStateCount();
int rowCount = Math.addExact(eventCount, Math.addExact(rollbackCount, entityStateCount));
int logicalRowCount = Math.subtractExact(Math.addExact(events.logicalSize(), Math.addExact(rollbackCount, entityStateCount)), state.getLocalOverlapCount(eventCount));
receipt = new ClickHouseBatchReceipt(identity.getProducerSequence(), identity.getBatchId(), rowCount, logicalRowCount);
return receipt;
}
boolean isPublished() {
ensureOpen();
return published;
}
void markPublished() {
ensureOpen();
published = true;
}
ClickHouseBatchIdentity getIdentity() {
return identity;
}
int size() {
ensureOpen();
return Math.addExact(events.size(), Math.addExact(state.getRollbackCount(), state.getEntityStateCount()));
}
InputStream openRows() {
if (receipt == null) {
throw new IllegalStateException("ClickHouse write batch is not sealed");
}
return events.openRows();
}
Checkpoint checkpoint() {
ensureOpen();
if (receipt != null) {
throw new IllegalStateException("ClickHouse write batch is already sealed");
}
return new Checkpoint(events.checkpoint(), state.checkpoint());
}
void restore(Checkpoint checkpoint) {
ensureOpen();
Objects.requireNonNull(checkpoint, "checkpoint");
if (receipt != null) {
throw new IllegalStateException("ClickHouse write batch is already sealed");
}
events.restore(checkpoint.events);
state.restore(checkpoint.state);
}
@Override
public void close() {
if (!closed) {
closed = true;
events.close();
state.close();
}
}
private void ensureOpen() {
if (closed) {
throw new IllegalStateException("ClickHouse write batch is closed");
}
}
static final class Checkpoint {
private final ClickHouseEventBatch.Checkpoint events;
private final ClickHouseStateBatch.Checkpoint state;
private Checkpoint(ClickHouseEventBatch.Checkpoint events, ClickHouseStateBatch.Checkpoint state) {
this.events = events;
this.state = state;
}
}
}
src/main/java/net/coreprotect/database/clickhouse/ClickHouseWriterRegistration.java
+258
View File
@@ -0,0 +1,258 @@
package net.coreprotect.database.clickhouse;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
import java.nio.channels.OverlappingFileLockException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Objects;
import java.util.UUID;
final class ClickHouseWriterRegistration implements AutoCloseable {
private static final String WRITER_FILE = ".clickhouse-writer";
private static final int MAX_WRITER_ID_BYTES = 64;
private final ClickHouseJdbc jdbc;
private final String table;
private final UUID datasetId;
private final UUID producerId;
private final Path writerFile;
private UUID writerId;
private FileChannel writerChannel;
private FileLock writerLock;
private boolean owned;
private boolean closed;
ClickHouseWriterRegistration(ClickHouseJdbc jdbc, String database, String prefix, UUID datasetId, UUID producerId, Path controlDirectory) {
this.jdbc = Objects.requireNonNull(jdbc, "jdbc");
table = ClickHouseIdentifiers.qualified(database, prefix + "writer_registration");
this.datasetId = Objects.requireNonNull(datasetId, "datasetId");
this.producerId = Objects.requireNonNull(producerId, "producerId");
writerFile = Objects.requireNonNull(controlDirectory, "controlDirectory").resolve(WRITER_FILE);
}
synchronized void acquire() throws SQLException {
ensureOpen();
if (owned) {
return;
}
acquireLocalWriter();
Registration registration = read();
if (registration == null) {
try {
insert();
}
catch (SQLException insertionFailure) {
try {
registration = read();
}
catch (SQLException reconciliationFailure) {
insertionFailure.addSuppressed(reconciliationFailure);
throw insertionFailure;
}
if (registration == null) {
throw insertionFailure;
}
}
if (registration == null) {
registration = read();
}
}
if (registration == null) {
throw new SQLException("ClickHouse writer registration is not visible after creation");
}
requireRegistration(registration);
owned = true;
}
synchronized void verifyOwned() throws SQLException {
ensureOpen();
requireOwned();
Registration registration = read();
if (registration == null) {
throw new OwnershipException("ClickHouse writer registration is not visible");
}
requireRegistration(registration);
}
private void requireRegistration(Registration registration) throws SQLException {
if (!datasetId.equals(registration.datasetId) || !producerId.equals(registration.producerId)) {
throw new OwnershipException("ClickHouse writer registration belongs to a different storage identity");
}
if (!writerId.equals(registration.writerId)) {
throw new OwnershipException("ClickHouse dataset is registered to a different CoreProtect installation");
}
}
synchronized void ensureOwned() throws SQLException {
ensureOpen();
requireOwned();
}
@Override
public synchronized void close() throws SQLException {
if (closed) {
return;
}
IOException failure = null;
try {
if (writerLock != null) {
writerLock.release();
}
}
catch (IOException exception) {
failure = exception;
}
try {
if (writerChannel != null) {
writerChannel.close();
}
}
catch (IOException exception) {
if (failure == null) {
failure = exception;
}
else {
failure.addSuppressed(exception);
}
}
finally {
writerLock = null;
writerChannel = null;
owned = false;
closed = true;
}
if (failure != null) {
throw new SQLException("Failed to close the local ClickHouse writer registration", failure);
}
}
private void acquireLocalWriter() throws SQLException {
if (writerLock != null && writerLock.isValid()) {
return;
}
try {
Files.createDirectories(writerFile.toAbsolutePath().getParent());
writerChannel = FileChannel.open(writerFile, StandardOpenOption.CREATE, StandardOpenOption.READ, StandardOpenOption.WRITE);
try {
writerLock = writerChannel.tryLock();
}
catch (OverlappingFileLockException exception) {
throw new SQLException("This CoreProtect installation already has an active ClickHouse writer", exception);
}
if (writerLock == null) {
throw new SQLException("This CoreProtect installation already has an active ClickHouse writer");
}
writerId = readOrCreateWriterId();
}
catch (IOException exception) {
throw new SQLException("Failed to open the local ClickHouse writer registration", exception);
}
}
private UUID readOrCreateWriterId() throws IOException, SQLException {
long size = writerChannel.size();
if (size == 0) {
UUID id = UUID.randomUUID();
byte[] encoded = id.toString().getBytes(StandardCharsets.US_ASCII);
writerChannel.position(0);
writeFully(writerChannel, ByteBuffer.wrap(encoded));
writerChannel.truncate(encoded.length);
writerChannel.force(true);
return id;
}
if (size > MAX_WRITER_ID_BYTES) {
throw new SQLException("Local ClickHouse writer registration is invalid");
}
ByteBuffer encoded = ByteBuffer.allocate((int) size);
writerChannel.position(0);
while (encoded.hasRemaining()) {
if (writerChannel.read(encoded) < 0) {
throw new SQLException("Local ClickHouse writer registration is incomplete");
}
}
String value = new String(encoded.array(), StandardCharsets.US_ASCII).trim();
try {
UUID id = UUID.fromString(value);
if (!id.toString().equalsIgnoreCase(value)) {
throw new IllegalArgumentException("Non-canonical UUID");
}
return id;
}
catch (IllegalArgumentException exception) {
throw new SQLException("Local ClickHouse writer registration is invalid", exception);
}
}
private Registration read() throws SQLException {
String sql = "SELECT dataset_id,producer_id,writer_id FROM " + table
+ " ORDER BY registration_order,writer_id LIMIT 1";
try (Connection connection = jdbc.openConnection(); PreparedStatement statement = connection.prepareStatement(sql); ResultSet resultSet = statement.executeQuery()) {
if (!resultSet.next()) {
return null;
}
return new Registration(UUID.fromString(resultSet.getString(1)), UUID.fromString(resultSet.getString(2)), UUID.fromString(resultSet.getString(3)));
}
}
private void insert() throws SQLException {
String sql = "INSERT INTO " + table
+ " (dataset_id,producer_id,writer_id,registered_at) VALUES (?,?,?,now64(3, 'UTC'))";
try (Connection connection = jdbc.openConnection(); PreparedStatement statement = connection.prepareStatement(sql)) {
statement.setObject(1, datasetId);
statement.setObject(2, producerId);
statement.setObject(3, writerId);
statement.execute();
}
}
private void requireOwned() throws SQLException {
if (!owned || writerLock == null || !writerLock.isValid()) {
throw new OwnershipException("ClickHouse writer registration is not owned by this CoreProtect installation");
}
}
private void ensureOpen() {
if (closed) {
throw new IllegalStateException("ClickHouse writer registration is closed");
}
}
private static void writeFully(FileChannel channel, ByteBuffer data) throws IOException {
while (data.hasRemaining()) {
channel.write(data);
}
}
private static final class Registration {
private final UUID datasetId;
private final UUID producerId;
private final UUID writerId;
private Registration(UUID datasetId, UUID producerId, UUID writerId) {
this.datasetId = datasetId;
this.producerId = producerId;
this.writerId = writerId;
}
}
static final class OwnershipException extends SQLException {
private static final long serialVersionUID = 1L;
private OwnershipException(String message) {
super(message);
}
}
}
src/main/java/net/coreprotect/database/logger/BlockBreakLogger.java
+4 -4
View File
@@ -1,6 +1,5 @@
package net.coreprotect.database.logger;
import java.sql.PreparedStatement;
import java.util.List;
import java.util.Locale;
@@ -12,6 +11,8 @@ import net.coreprotect.CoreProtect;
import net.coreprotect.bukkit.BukkitAdapter;
import net.coreprotect.config.Config;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.Database;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.BlockStatement;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.event.CoreProtectPreLogEvent;
@@ -20,7 +21,6 @@ import net.coreprotect.thread.CacheHandler;
import net.coreprotect.utility.BlockTypeUtils;
import net.coreprotect.utility.MaterialUtils;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.ErrorReporter;
public class BlockBreakLogger {
@@ -28,7 +28,7 @@ public class BlockBreakLogger {
throw new IllegalStateException("Database class");
}
public static void log(PreparedStatement preparedStmt, int batchCount, String user, Location location, int type, int data, List<Object> meta, String blockData, String overrideData) {
public static void log(ConsumerWriteBatch preparedStmt, int batchCount, String user, Location location, int type, int data, List<Object> meta, String blockData, String overrideData) {
try {
Material checkType = MaterialUtils.getType(type);
String blockKey = BlockTypeUtils.getBlockDataKey(blockData);
@@ -91,7 +91,7 @@ public class BlockBreakLogger {
BlockStatement.insert(preparedStmt, batchCount, time, userId, wid, x, y, z, internalType, data, meta, blockData, LookupActions.BLOCK_BREAK, 0);
}
catch (Exception e) {
ErrorReporter.report(e);
Database.handleWriteFailure(e);
}
}
src/main/java/net/coreprotect/database/logger/BlockPlaceLogger.java
+4 -4
View File
@@ -1,6 +1,5 @@
package net.coreprotect.database.logger;
import java.sql.PreparedStatement;
import java.util.List;
import java.util.Locale;
@@ -13,6 +12,8 @@ import net.coreprotect.CoreProtect;
import net.coreprotect.bukkit.BukkitAdapter;
import net.coreprotect.config.Config;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.Database;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.BlockStatement;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.event.CoreProtectPreLogEvent;
@@ -21,7 +22,6 @@ import net.coreprotect.thread.CacheHandler;
import net.coreprotect.utility.BlockTypeUtils;
import net.coreprotect.utility.MaterialUtils;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.ErrorReporter;
public class BlockPlaceLogger {
@@ -29,7 +29,7 @@ public class BlockPlaceLogger {
throw new IllegalStateException("Database class");
}
public static void log(PreparedStatement preparedStmt, int batchCount, String user, BlockState block, int replacedType, int replacedData, Material forceType, int forceData, boolean force, List<Object> meta, String blockData, String replaceBlockData) {
public static void log(ConsumerWriteBatch preparedStmt, int batchCount, String user, BlockState block, int replacedType, int replacedData, Material forceType, int forceData, boolean force, List<Object> meta, String blockData, String replaceBlockData) {
try {
Material type = block.getType();
if (blockData == null && (forceType == null || (!forceType.equals(Material.WATER)) && (!forceType.equals(Material.LAVA)))) {
@@ -134,7 +134,7 @@ public class BlockPlaceLogger {
BlockStatement.insert(preparedStmt, batchCount, time, userId, wid, x, y, z, internalType, data, meta, blockData, LookupActions.BLOCK_PLACE, 0);
}
catch (Exception e) {
ErrorReporter.report(e);
Database.handleWriteFailure(e);
}
}
src/main/java/net/coreprotect/database/logger/ChatLogger.java
+4 -4
View File
@@ -1,14 +1,14 @@
package net.coreprotect.database.logger;
import java.sql.PreparedStatement;
import java.util.Locale;
import org.bukkit.Location;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.Database;
import net.coreprotect.database.statement.ChatStatement;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.ErrorReporter;
public class ChatLogger {
@@ -16,7 +16,7 @@ public class ChatLogger {
throw new IllegalStateException("Database class");
}
public static void log(PreparedStatement preparedStmt, int batchCount, long time, Location location, String user, String message) {
public static void log(ConsumerWriteBatch preparedStmt, int batchCount, long time, Location location, String user, String message) {
try {
if (ConfigHandler.isBlacklisted(user)) {
return;
@@ -29,7 +29,7 @@ public class ChatLogger {
ChatStatement.insert(preparedStmt, batchCount, time, userId, wid, x, y, z, message);
}
catch (Exception e) {
ErrorReporter.report(e);
Database.handleWriteFailure(e);
}
}
src/main/java/net/coreprotect/database/logger/CommandLogger.java
+4 -4
View File
@@ -1,6 +1,5 @@
package net.coreprotect.database.logger;
import java.sql.PreparedStatement;
import java.util.Locale;
import org.bukkit.Bukkit;
@@ -9,11 +8,12 @@ import org.bukkit.Location;
import net.coreprotect.CoreProtect;
import net.coreprotect.config.Config;
import net.coreprotect.config.ConfigHandler;
import net.coreprotect.database.Database;
import net.coreprotect.database.ConsumerWriteBatch;
import net.coreprotect.database.statement.CommandStatement;
import net.coreprotect.database.statement.UserStatement;
import net.coreprotect.event.CoreProtectPreLogEvent;
import net.coreprotect.utility.WorldUtils;
import net.coreprotect.utility.ErrorReporter;
public class CommandLogger {
@@ -21,7 +21,7 @@ public class CommandLogger {
throw new IllegalStateException("Database class");
}
public static void log(PreparedStatement preparedStmt, int batchCount, long time, Location location, String user, String message) {
public static void log(ConsumerWriteBatch preparedStmt, int batchCount, long time, Location location, String user, String message) {
try {
if (ConfigHandler.isBlacklisted(user)) {
return;
@@ -48,7 +48,7 @@ public class CommandLogger {
CommandStatement.insert(preparedStmt, batchCount, time, userId, wid, x, y, z, message);
}
catch (Exception e) {
ErrorReporter.report(e);
Database.handleWriteFailure(e);
}
}
Loaded 100 of 152 files, more files were not shown because too many files have changed in this diff. Show more