Gardbase Golang SDK

I’m finally implementing queries. I wrote a query builder interface for it to feel modern and comfortable for users and I’m trying to correctly handle encrypted indexes. I implemented the entire query operation client-side.
I also refactored SCAN’s inputs and outputs and how tokens are encrypted and generated.

I added a DELETE route (which works by soft-deleting items and tagging them with a 30 days TTL) and also a RECOVER route to recover deleted objects before the 30 days.
The DELETE operation soft deletes the object (by changing its status), but deletes its indexes (so that it can’t be searched), so RECOVER can only prevent the final object deletion, but won’t restore the indexes. In the future, they could be restored either by going through the enclave or by re-encrypting the indexes client-side.

I just found out that the JSON encode function can actually handle []byte slices (lol), so I just replaced all B64-encoded strings in requests (both standard client-server and server-enclave requests) with []byte and remove all the unnecessary B64 encoding/decoding. All tests are still passing!!!

I refactored how indexes are structured and, therefore, how schemas are initialized and handled under the hood. I made users able to create composite HASH+RANGE composite indexes (inspired by DynamoDB design) (before this, users just couldn’t query for multiple indexes at the same time without manual result aggregation).
Now, at schema creation, users have to pass both a gardb.Model struct, which includes the basic schema definitions, and a new gardb.Indexes that contains all the indexes that will be created alongside the object itself in DynamoDB.

After the big PUT rework, I continued implementing the update functionality both with the PUT method and with a new UPDATE method (which, under the hood, just combines a GET with a PUT with some safety enhancements, since everything is encrypted client-side anyways). I implemented the actual UPDATE method and I had to fix some problems with versioning. Now all tests pass again (focus on tests 07 and 08)

I did a complete refactoring of how the PUT operation works.
Before, the flow was similar for small and large objects:

• Client sends object’s metadata
• Server creates metadata document in DynamoDB with TTL, sends back a presigned PUT url (directing to S3 if it’s a large object, or to another route if it’s a small object)
• Client uploads encrypted blob
• Server removes TTL
  However, this common approach was not optimized, as most of the objects will be small in size and could be uploaded with a single request. Therefore, I changed it to:
  Small objects:
• Client sends everything to Server
• Server uploads everything to DynamoDB
  Large objects:
• Client asks for a presigned put url (S3)
• Server generates the url
• Client uploads object
• Client sends to Server all the object’s metadata

Also, removing the PENDING state enables me to handle object versioning in an easier way, as I only need to put an object with a different version now to update it, while before I had to work with multiple pending upload requests and had to filter objects by state. This enables me to complete the PUT route (I wanted to add the possibility to replace an existing object through PUT) and fully create an UPDATE route.

I was able to make all tests pass after the PUT rework

I completely refactored how schemas are created. Now they use generics to directly reference the struct from which they are generated, so that users now have compile-time type safety. Also, this way, I refactored the way GET and SCAN operations return their results, replacing unmarshalling data into a pointer parameter to just returning an array of structs.

First of all, I refactored how the SCAN operation works so that now paginating the results is way more straightforward (I took inspiration from how DynamoDB does it). I also added a lot more integration tests, even for edge cases and I’m fixing everything to make them all pass little by little. I also fixed a very bad issue with time values and integers that was happening while unmarshalling the unwrapped data into the object pointer after GET or SCAN and I fixed the SDK’s behavior when scanning empty tables or with 0 as limit.

I had to fix some issues with the GET route and I also added a SCAN route and all the tests for it. I reorganized the object API structure on the backend, so I had to make some changes to the SDK too. Here are the changes for the API structure (inspired by DynamoDB operations):

• POST “/:table/:id” -> POST “/put”
• GET “/:table/:id” -> POST “/get”
• POST “/:table/scan” -> POST “/scan”
  Right now all the tests are passing!

After refactoring how the objects are stored on the backend (S3 -> hybrid DynamoDB + S3), I had to rewrite some parts of the API client so that everything connects as it should. Something is still not working, so the GET test is currently failing

After refactoring, I went through all the unit and integration tests again to fix them. I first had to completely rewrite the tests, then I had to debug a little to fix all those minor code bugs I did while refactoring (e.g., I was writing all API keys to the tenants table instead of the keys table…). The screenshots show all the tests passing… again.

I spent a LOT of time figuring out how the SDK should work. First of all, since I refactored the entire Gardbase’s API structure, I had to refactor the PUT and GET flows.
I then refactored the relationship between the Client and the Schema objects, moving Schema to the gardb package and only leaving the Field struct in the schema package: this ensures that I can easily integrate the Schema struct within the gardb package, and leads to the biggest change I made - I moved the GET and PUT methods from the Client struct to the Schema struct. I figured that the Schema struct should’ve had the responsibility of handling PUT and GET, and not the Client (since this SDK is more similar to a mongoose-like interface than to a DynamoDB-like one).
I consequently had to change how the client and the schemas are initialized and the whole validation flow.
I also added a simple cache to persist table hashes and not having to call the backend to hash table names each time the client and schemas are initialized.

I’m still working on making all the basic integration tests pass for the SDK and today I worked on the GET operation. I noticed there was an error linked to encryption, so I re-analyzed and debugged the flow to find out my crypto library wasn’t working as expected.

I had to debug and rewrite the section that integrated the library, update it and now it seems to work!

I created some integration tests to test the actual working APIs and check that everything works as expected… and it doesn’t, obviously. So I spent some time figuring out what was wrong, I rewrote some parts of the SDK that handled communication with the enclave and fixed some bugs with the actual API handling (like how multitenancy was handled client-side).

I’ve been adding more tests to the SDK and I’m trying to solve some issues related to the vsock communication between the parent application (my server) and the enclave app. Also, I’ve been enhancing the way errors are structured in order for users to have little to no trouble figuring out what is actually wrong.