Gardbase Golang SDK

I tried out the project one last time and I fixed all the imports and how the package was managed. I also changed some parts of the README to make sure everything actually works fine. I also created a demo package to try to import everything externally and try if everything worked fine.

I added a new thorough readme with an API reference and some info about the project

I added way more tests in the SDK and fixed various minor bugs server-side:

• tenant tests
• scan tests
• edge cases tests
• large object tests

I worked on making more tests pass, specifically pagination (query) and concurrency ones. I refactored the way DEKs were generated and handled during PUTs and I also enhanced the SDK’s API layer. Now all the tests are passing

I spent some time fixing all the problems that came to light after I wrote the new tests, mostly regarding queries. I had to refactor how queries were built both client(sdk)-side and server-side, specifically focusing on how between queries were handled and completely adding a new manual ordering system.

I implemented some new integration tests (specifically regarding concurrency and edge-cases). I also started working on making all query tests work as expected: right now, the between operator is always returning 0 items and the pagination + ordering systems don’t work.

I’m setting up a whole new integration test suite that covers a lot more cases than the old tests in order to prepare gardbase for an actual first ship/release pretty soon. Obviously nothing is working good right now but this is why I’m implementing these new tests in the first place LOL

Through testing I noticed that hash+range queries weren’t working right, so I started debugging. Indeed, there were inconsistencies in how time ranges were encrypted, so values normalization and OPE didn’t really work as expected and queries gave wrong results. To find the underlying issue, I had to manually debug and check tokens (bytes…) before and after encryption, study the results and locate where in the code something was going wrong. I had to rewrite the values normalization function and had to fix the current (temporary) OPE implementation to prevent uint64 overflow and result inconsistencies and now everything works good!

After implementing the simplest query operation (hash only indexes), I’m now working on implementing range queries (GT, LT, GTE, LTE, BETWEEN) by comparing token’s bytes lexicographically. Indexes are composed of 3 parts: the deterministically encrypted hash value, the range value encrypted with OPE and the bytes representation of object ID. This index structure allows me to search both for hash value only and hash+range value. I also changed the structure of BETWEEN requests to make them as clear and easy to understand to other people as I can.
The system doesn’t quite work yet, I’m still solving some issues (tests still fail).

After implementing the initial querying system, I spent some time debugging tests to fix everything. I had to fix how the object count (returned by DynamoDB itself) was handled by the backend and then returned to the SDK, I had to update permissions for the EC2 (I totally forgot to add the permissions to perform BatchGetItem to get all objects from indexes in queries), and I had to add a filter to the SCAN operation in order to exclude all objects marked as deleted. Now all tests are passing!

I’m finally implementing queries. I wrote a query builder interface for it to feel modern and comfortable for users and I’m trying to correctly handle encrypted indexes. I implemented the entire query operation client-side.
I also refactored SCAN’s inputs and outputs and how tokens are encrypted and generated.

I added a DELETE route (which works by soft-deleting items and tagging them with a 30 days TTL) and also a RECOVER route to recover deleted objects before the 30 days.
The DELETE operation soft deletes the object (by changing its status), but deletes its indexes (so that it can’t be searched), so RECOVER can only prevent the final object deletion, but won’t restore the indexes. In the future, they could be restored either by going through the enclave or by re-encrypting the indexes client-side.

I just found out that the JSON encode function can actually handle []byte slices (lol), so I just replaced all B64-encoded strings in requests (both standard client-server and server-enclave requests) with []byte and remove all the unnecessary B64 encoding/decoding. All tests are still passing!!!

I refactored how indexes are structured and, therefore, how schemas are initialized and handled under the hood. I made users able to create composite HASH+RANGE composite indexes (inspired by DynamoDB design) (before this, users just couldn’t query for multiple indexes at the same time without manual result aggregation).
Now, at schema creation, users have to pass both a gardb.Model struct, which includes the basic schema definitions, and a new gardb.Indexes that contains all the indexes that will be created alongside the object itself in DynamoDB.

After the big PUT rework, I continued implementing the update functionality both with the PUT method and with a new UPDATE method (which, under the hood, just combines a GET with a PUT with some safety enhancements, since everything is encrypted client-side anyways). I implemented the actual UPDATE method and I had to fix some problems with versioning. Now all tests pass again (focus on tests 07 and 08)

I did a complete refactoring of how the PUT operation works.
Before, the flow was similar for small and large objects:

• Client sends object’s metadata
• Server creates metadata document in DynamoDB with TTL, sends back a presigned PUT url (directing to S3 if it’s a large object, or to another route if it’s a small object)
• Client uploads encrypted blob
• Server removes TTL
  However, this common approach was not optimized, as most of the objects will be small in size and could be uploaded with a single request. Therefore, I changed it to:
  Small objects:
• Client sends everything to Server
• Server uploads everything to DynamoDB
  Large objects:
• Client asks for a presigned put url (S3)
• Server generates the url
• Client uploads object
• Client sends to Server all the object’s metadata

Also, removing the PENDING state enables me to handle object versioning in an easier way, as I only need to put an object with a different version now to update it, while before I had to work with multiple pending upload requests and had to filter objects by state. This enables me to complete the PUT route (I wanted to add the possibility to replace an existing object through PUT) and fully create an UPDATE route.

I was able to make all tests pass after the PUT rework

I completely refactored how schemas are created. Now they use generics to directly reference the struct from which they are generated, so that users now have compile-time type safety. Also, this way, I refactored the way GET and SCAN operations return their results, replacing unmarshalling data into a pointer parameter to just returning an array of structs.

First of all, I refactored how the SCAN operation works so that now paginating the results is way more straightforward (I took inspiration from how DynamoDB does it). I also added a lot more integration tests, even for edge cases and I’m fixing everything to make them all pass little by little. I also fixed a very bad issue with time values and integers that was happening while unmarshalling the unwrapped data into the object pointer after GET or SCAN and I fixed the SDK’s behavior when scanning empty tables or with 0 as limit.

I had to fix some issues with the GET route and I also added a SCAN route and all the tests for it. I reorganized the object API structure on the backend, so I had to make some changes to the SDK too. Here are the changes for the API structure (inspired by DynamoDB operations):

• POST “/:table/:id” -> POST “/put”
• GET “/:table/:id” -> POST “/get”
• POST “/:table/scan” -> POST “/scan”
  Right now all the tests are passing!

After refactoring how the objects are stored on the backend (S3 -> hybrid DynamoDB + S3), I had to rewrite some parts of the API client so that everything connects as it should. Something is still not working, so the GET test is currently failing

After refactoring, I went through all the unit and integration tests again to fix them. I first had to completely rewrite the tests, then I had to debug a little to fix all those minor code bugs I did while refactoring (e.g., I was writing all API keys to the tenants table instead of the keys table…). The screenshots show all the tests passing… again.

I spent a LOT of time figuring out how the SDK should work. First of all, since I refactored the entire Gardbase’s API structure, I had to refactor the PUT and GET flows.
I then refactored the relationship between the Client and the Schema objects, moving Schema to the gardb package and only leaving the Field struct in the schema package: this ensures that I can easily integrate the Schema struct within the gardb package, and leads to the biggest change I made - I moved the GET and PUT methods from the Client struct to the Schema struct. I figured that the Schema struct should’ve had the responsibility of handling PUT and GET, and not the Client (since this SDK is more similar to a mongoose-like interface than to a DynamoDB-like one).
I consequently had to change how the client and the schemas are initialized and the whole validation flow.
I also added a simple cache to persist table hashes and not having to call the backend to hash table names each time the client and schemas are initialized.

I’m still working on making all the basic integration tests pass for the SDK and today I worked on the GET operation. I noticed there was an error linked to encryption, so I re-analyzed and debugged the flow to find out my crypto library wasn’t working as expected.

I had to debug and rewrite the section that integrated the library, update it and now it seems to work!

I created some integration tests to test the actual working APIs and check that everything works as expected… and it doesn’t, obviously. So I spent some time figuring out what was wrong, I rewrote some parts of the SDK that handled communication with the enclave and fixed some bugs with the actual API handling (like how multitenancy was handled client-side).

I’ve been adding more tests to the SDK and I’m trying to solve some issues related to the vsock communication between the parent application (my server) and the enclave app. Also, I’ve been enhancing the way errors are structured in order for users to have little to no trouble figuring out what is actually wrong.