AGEFreighter 2.2.0をリリースしました
rio
PostgreSQL 19のリリースが迫ってきたので、Beta 3で動作を確認すると共に、プロパティグラフによるグラフデータが扱えるようになるので、Apache AGEとは別に、SQL/PGQをサポートするようにしました。クエリ言語はCypherクエリとは別ですが、通常のSQLと組み合わせるのであればCypherよりは楽なので、将来的にはAGEが不要になるかもしれません。
https://github.com/rioriost/agefreighter/releases/tag/v2.2.0
さて、これとは別に、本番環境を模したデータサイズのNeo4jのグラフデータを移行する検証を、4日ほどかけて実施しました。移行先は、Azure Database for PostgreSQL Flexible ServerのPostgreSQL 18 + Apache AGE 1.7です。
詳細はP3 production-simulation resultを見てもらうとして、1.6億ノード、4.0億エッジのグラフデータの移行が可能なことを確認できました。
| Source | Run | Load result | Recovery evidence | Canonical verification |
|---|---|---|---|---|
| Neo4j 4.4.48 | Clean | 560,000,000 committed in 3:55:27; zero rejects | Not applicable | 5,600/5,600 ranges and root match; digest 2:13:21 |
| Neo4j 4.4.48 | Recovery | 560,000,000 committed; zero rejects | SIGTERM at 140,920,000 rows; Neo4j restart at 229,300,000 rows; same job resumed after both | 5,600/5,600 ranges and root match; digest 1:58:09 |
| Neo4j 5.26.30 | Clean | 560,000,000 committed in 5:37:01; zero rejects | Not applicable | 5,600/5,600 ranges and root match; digest 2:14:14 |
| Neo4j 5.26.30 | Recovery | 560,000,000 committed; zero rejects | PostgreSQL connectivity fault at 344,700,000 rows; loader-VM reboot after 433,740,000 rows; same job resumed after both | 5,600/5,600 ranges and root match; digest 2:00:14 |
RunがRecoveryになっているパターンは、途中で接続が失われたり、AGEFreighterが動いているVMを強制終了したりといった事象があっても、移行を正しく行えるかを検証したものです。AGEFreighter 1.xではこのパターンに弱く、2.xはその改善も含めたものとしました。
次は、Neo4j + APOCを移行できるようにする機能を追加したいと思います。
In English
With the PostgreSQL 19 release approaching, I’ve verified that it works with Beta 3. Since PostgreSQL 19 will also support graph data using property graphs, I’ve added support for SQL/PGQ alongside Apache AGE.
The query syntax is different from Cypher, but when working together with regular SQL, SQL/PGQ is actually easier to use than Cypher. So, in the future, we may no longer need AGE at all.
https://github.com/rioriost/agefreighter/releases/tag/v2.2.0
Separately from this, I spent about four days conducting a migration validation using Neo4j graph data at a scale designed to simulate a production environment.
The migration target was Azure Database for PostgreSQL Flexible Server running PostgreSQL 18 with Apache AGE 1.7.
For the details, please refer to the P3 production-simulation results. In short, I confirmed that it is possible to migrate graph data at a scale of 160 million nodes and 400 million edges.
| Source | Run | Load result | Recovery evidence | Canonical verification |
|---|---|---|---|---|
| Neo4j 4.4.48 | Clean | 560,000,000 committed in 3:55:27; zero rejects | Not applicable | 5,600/5,600 ranges and root match; digest 2:13:21 |
| Neo4j 4.4.48 | Recovery | 560,000,000 committed; zero rejects | SIGTERM at 140,920,000 rows; Neo4j restart at 229,300,000 rows; same job resumed after both | 5,600/5,600 ranges and root match; digest 1:58:09 |
| Neo4j 5.26.30 | Clean | 560,000,000 committed in 5:37:01; zero rejects | Not applicable | 5,600/5,600 ranges and root match; digest 2:14:14 |
| Neo4j 5.26.30 | Recovery | 560,000,000 committed; zero rejects | PostgreSQL connectivity fault at 344,700,000 rows; loader-VM reboot after 433,740,000 rows; same job resumed after both | 5,600/5,600 ranges and root match; digest 2:00:14 |
The runs labeled as “Recovery” were designed to verify that the migration could still complete correctly even when failures occurred during the process, such as a lost connection or forcibly shutting down the VM running AGEFreighter.
AGEFreighter 1.x was relatively weak in this area, so improving recovery and resiliency was one of the key goals of the 2.x release.
Next, I’d like to add support for migrating Neo4j environments that use APOC.
