Gardbase

I updated the readme with new updated docs

I added way more tests in the SDK and fixed various minor bugs server-side:

• tenant tests
• scan tests
• edge cases tests
• large object tests

I worked on making more tests pass, specifically pagination and concurrency ones. I had to refactor the rate limiter middleware, how DEKs were built (I was using one nonce for the entire DEKs generation session, instead of one nonce per DEK) and I enhanced the layer of communication between the backend and the enclave (the vsocket structure).

I worked on fixing all the new problems that came up after writing new SDK tests specifically regarding queries. I had to completely rework the query building system, how indexes were handled, how results were ordered (I had to implement a whole new manual ordering system), and how between queries worked. Now everything looks good, except for a little (I hope) consistency issue that’s still breaking one of the tests.

Through testing I noticed that hash+range queries weren’t working right, so I started debugging. Indeed, there were inconsistencies in how time ranges were encrypted, so values normalization and OPE didn’t really work as expected and queries gave wrong results. To find the underlying issue, I had to manually debug and check tokens (bytes…) before and after encryption, study the results and locate where in the code something was going wrong. I had to rewrite the values normalization function and had to fix the current (temporary) OPE implementation to prevent uint64 overflow and result inconsistencies and now everything works good!

After implementing the simplest query operation (hash only indexes), I’m now working on implementing range queries (GT, LT, GTE, LTE, BETWEEN) by comparing token’s bytes lexicographically. Indexes are composed of 3 parts: the deterministically encrypted hash value, the range value encrypted with OPE and the bytes representation of object ID. This index structure allows me to search both for hash value only and hash+range value.
The system doesn’t quite work yet, I’m still solving some issues (tests still fail).

After implementing the initial querying system, I spent some time debugging tests to fix everything. I had to fix how the object count (returned by DynamoDB itself) was handled by the backend and then returned to the SDK, I had to update permissions for the EC2 (I totally forgot to add the permissions to perform BatchGetItem to get all objects from indexes in queries), and I had to add a filter to the SCAN operation in order to exclude all objects marked as deleted. Now all tests (on the SDK) are passing!

I’ve been trying to implement queries. They work by querying the indexes table with a client-side encrypted token, then reverse get the object through the object ID reference saved in the index document. I implemented the first type of query (hash-only index, plain equality), even though it still doesn’t completely work.
I also refactored SCAN’s inputs and outputs and how token are handled server-side (now a byte representation of the object ID is appended to avoid PK-SK pairs duplication issues).

I added a DELETE route (which works by soft-deleting items and tagging them with a 30 days TTL) and also a RECOVER route to recover deleted objects before the 30 days.
The DELETE operation soft deletes the object (by changing its status), but deletes its indexes (so that it can’t be searched), so RECOVER can only prevent the final object deletion, but won’t restore the indexes. In the future, they could be restored either by going through the enclave or by re-encrypting the indexes client-side.

I just found out that the JSON encode function can actually handle []byte slices (lol), so I just replaced all B64-encoded strings in requests (both standard client-server and server-enclave requests) with []byte and remove all the unnecessary B64 encoding/decoding. All tests are still passing!!!

I started actually working on the indexes and I refactored how they will work. Since DynamoDB and the zero-trust principle wouldn’t have made users able to query for multiple indexes at the same time without manual aggregation, I had to change how indexes are created in the first place to enable the creation of DynamoDB-inspired composite Hash(fixed length DET ENC)+Range(OPE->todo make it safe) indexes to grant a little more usability.

After doing completely rework of the PUT operation, I used the SDK’s integration tests to make sure everything was working fine and, obviously, it wasn’t. However I only had to fix some minor issues with versioning and dynamodb operations, and now all tests pass! (look at 08 and 09)

First of all, sorry for the 6h devlog, I couldn’t have devlogged sooner and I’ll explain the reasons here.
I was trying to add a way to update existing objects and I wanted to take inspiration from how DynamoDB does it (PutItem to create or replace existing objects, UpdateItem to partially update objects). To enable versioning, I completely refactored how the PUT operation works, also optimizing how lightweight objects are uploaded (before, common flow of: client sends metadata -> server creates pending object and sends a presigned put url -> client uploads the object; now, for lightweight objects, everything is done in one backend call, while for large objects the client first asks for a presigned put url, then confirms the upload to the server and only later the metadata is stored in DynamoDB).
However, while I was figuring out how indexes should be updated on an object update, I noticed that they were currently encrypted (with det. enc.) using the object’s DEK as key. Obviously, this completely broke how index should work, since a user should be able to det. encrypt an index token and search for it across the entire table. To solve this problem, I had to create a new TableConfig table, which stores, among other table metadata, the WrappedIEK field, which represents a KMS-wrapped table-wide Index Encryption Key to use for consistent and deterministic index encryption.
This way, I solved both the versioning issue and the index encryption consistency issue.
To complete the PUT operation refactoring, I refactored how indexes are stored, adding a DynamoDB Global Secondary Index to enable a new Object ID access pattern (I needed to search for every index of a specific object).

Finally, after all these problems, I completed the refactoring, enabling users to update their objects and indexes, and in the picture attached there’s the proof of it passing all tests!

I added more integration tests to the SDK, so I spent some time fixing all the backend-side issues with the existing code. I also refactored the SCAN operation so that it correctly implements pagination now (heavily inspired by DynamoDB).

I had to fix some issues with the GET route and I also added a SCAN route and all the tests for it on the SDK. I reorganized the object API structure, with these changes (inspired by DynamoDB operations):

• POST “/:table/:id” -> POST “/put”
• GET “/:table/:id” -> POST “/get”
• POST “/:table/scan” -> POST “/scan”
  Right now all the integration tests on the SDK are passing!

I was thinking about adding some more operations like SCAN and then QUERY, but I came across a problem. In the GET operations, objects metadata was fetched from DynamoDB, then the server created a get presigned url for S3 and the client would have just downloaded the encrypted object from there. However, scaling up, this meant eventually creating hundreds of thousands of presigned url, creating both performance issues and security issues. So I decided to refactor how objects are stored: now, the client first declares the encrypted blob’s size; if it’s a large object (>100KB) it the server will send a put presigned url for s3; if it’s a small object (<100KB), then it will generate the metadata in dynamodb and send a url for the upload of the blob to DynamoDB inline. This way, everything becomes way easier to scale, without creating very big performance or cost issues.

I’m still fixing how everything connects, so the GET operation is currently failing

After the big refactoring, I started fixing all the bugs I could find by debugging the Go SDK tests. I fixed a few bugs, like an error regarding how hashing was handled, or some stupid mistake like writing all API keys to the tenants table instead of their own table, but I managed to fix everything and now all the tests pass again!

I refactored the API (again…) and extracted all requests’ and responses’ structs into a different package, so that I can import everything from the Go SDK repo. I also refactored how health checks work and spent some time fixing problems with the infrastructure in terraform. I added the logic to validate tenants and api keys with a middleware for secure access and refactored S3 key generation for consistency.

I completed the API refactoring by re-organizing handlers in internal/handlers, so I clarified all the requests and responses structs and the crypto and enclaveproto packages accordingly. I also fixed an issue regarding KMS’s policy in the infrastructure’s configuration, making it safer.

I did a complete refactoring of the API structure, moving all the enclave functions from the enclave proxy (which, in fact, was not a proxy anymore) to a service-like struct that I can reuse across all handlers, making communicating with the enclave much simpler and more straightforward. While doing the refactoring, I also completed the table hash generation logic, making use of the shared session key with the enclave.
I also refactored how KMS is called, moving recurring logic to a service struct in order to make the code easier to read and less error-prone.

I refactored the object’s model to not include EncryptedTableName (since it can be easily referred from context) and changed how the PUT flow works. Specifically, I added an enclave handler to hash table names so that the client just needs to send the table name encrypted with the shared session key to the backend and nothing gets leaked in the process

I spent a few hours solving some structural issues and figuring out how to authenticate tenant for the application to work as expected. I reflected on how to handle key hierarchy and came out with a 4 level approach:

• Level 1: KMS Key (AWS-managed) - one key per environment, used to encrypt tenant master keys
• Level 2: Tenant Master Key (TMK) - Random 32 bytes per tenant, stored KMS-wrapped in DynamoDB, used to encrypt DEKs
• Level 3: Data Encryption Keys (DEKs) - Random 32 bytes per object, stored KMS-wrapped and TMK-wrapped in DynamoDB, used to encrypt actual object data
• Level 4: Data, encrypted with DEK
  I also added tenant registration and API key handling

I’ve been trying to implement how tenants are handled: so tenant registration, authentication, etc, following my application’s zero-trust model. I created a tenant config object in DynamoDB that stores tenant information such as encrypted master key (that, for now, I think will be provided to the user for requests authentication), a wrapped table salt, used to encrypt table names too, metadata like timestamps, version, IDs, key rotation information, recovery contacts. I’m trying to find the optimal way of handling tenant registration and then use through the SDK

I’m still working on making all the basic integration tests pass for the SDK and today I worked on the GET operation. I noticed there was an error linked to encryption, so I re-analyzed and debugged the flow to find out my crypto library wasn’t working as expected and wasn’t opening sealed DEKs decrypted through the enclave service and re-encrypted with the derived session key.
Now everything seems to work as expected!

I continued going through the SDK’s tests to verify if every route worked as expected and fixed some errors regarding the creation of objects. Now objects are successfully created as they should and those tests are passing! I had to fix how multitenancy was handled and a bug regarding the enclave’s attestation document.

I applied the same fix to both decrypt and session_unwrap handlers. Now they both call the enclave’s route for get_attestation and use the attestation for decrypting with KMS. Then they return KMS’s CiphertextForRecipient field to the enclave, that in turn decrypts it with OpenSSL and re-encrypts it with the session key/client ephemeral public key, so that the proxy NEVER sees the plaintext DEK.

Note: the code highlighted in the images is not the complete updated code - the full commits are:

• https://github.com/QodeSrl/Gardbase/commit/cf668e8b980c0de19d5c603989fb2d594dacb38c
• https://github.com/QodeSrl/Gardbase/commit/ca481d41fcd59ccdcc820fb4247debcbdcbe55bc
• https://github.com/QodeSrl/Gardbase/commit/f53e6bbe8be6b535288cb3a661bf636fcdfd6754
• https://github.com/QodeSrl/Gardbase/commit/a285a5043b43dcc934b606e827cccf7e53b5d0d5

I fixed the way the server handles key generation by moving the logic from the isolated enclave to the proxy, using the enclave’s cryptographic attestation document to guarantee zero-trust.

• I SPEND 5 EXCRUCIATING HOURS TESTING HOW THIS KMS ENCRYPTION SYSTEM WORKS BUT FINALLY I GOT IT!!!!! Credits to this super helpful answer on stackoverflow:
  https://stackoverflow.com/a/77985003/21306952 -

The proxy then sends the results to the enclave, which securely decrypts the new keys and re-encrypts them with a public key provided by the client, so that the keys are cyphertext to the proxy, but can be decrypted locally by the client.

The enclave-proxy system now finally works as it should!!!!!

While also working on the SDK I’m noticing a few errors regarding the vsock communication between the main server (the API) and the Enclave service running inside of it. Also, I’ve been trying to fix how the enclave handles calls to AWS KMS to generate and handle Data Encryption Keys (DEKs), so I’m trying to figure out how I can make calls like those from an isolated environment. I’m studying the official AWS Nitro Enclaves SDK (only written in,,, C 😭) to figure out how to do the same thing in Go.