Development of a new way of searching and finding files

Day 30 Authentication, Header and ZeroSSL understanding

What I did: improved protocol and transcribed to documentation

Time spent: 71.517


1 - This just like regular packets have a component that can be forged by external systems. However replay attacks are different then hold and send later attacks. For the first we're protected against. For the second (that is a message is intercepted and held so that it doesn't arrive and then some time in the future it's released, the system will accept (presuming the connection wasn't closed)). However that can be migitated at a higher level by having time sensitive dual exchange so a back and forth so that commands can't be held and released at a later time. Furthermore in the event that messages aren't being acknowledged the system can just terminate the 2SSL connection and when the server cleans up old connections that UUID will be wiped out marking that message (should it ever arrive) as invalid.

2 - Began setting up NX and setting up automated testing for Kyber.
- Reading how NX works again.
- Figuring out how to make the tests work and ensure code works in the current testing.
- The code and the testing is horrible and it's very annoying. It should be setup in require but it's currently using import/export which I believe works natively. There's alot of commented out code as well for Kyber. This is good thing to be cleaning up but it's still very frustrating.

Day 29 2SSL, PiB data transport and quantum key exchange protocol

What I did: Simplify protocol, review quantum protocol and review 2SSL transmission protocol to ensure data up to 1PiB can be transported.

Time spent: 69.333


1 - Deal with Green Geeks answer regarding 2SSL so we're stuck with 128 bit encryption for main site on Firefox....

2 - Dealing with version, security, replay attack and randomValue (avoiding issue with Unicode encoding issues) along with versioning

3 - Quantum Key distribution understanding

4 - Header redesign since each 2SSL connection uses a two password encryption which means that an individual message cracked doesn't leak data for any other data sent within the same channel. In the event that the 2SSL connection level derived key is cracked only that connect and no other connect is cracked.
This means to crack the connections would mean having to crack each connection stream (since each connection stream has it's own primary encryption and each packet is encrypted with a separate key that is unique to that packet.)

This means that TLS + 2SSL ensures 3 layers of encryption preventing leaking data. These first 16 bytes of the session are the session UUID and

5 - Planning replay resistance by tying UUID to the session so the same UUID can't be used. During a retransmission attempt we can request UUID's to be cleared.

6 - Using bloom filters as a probablistic data structures. Since these can be precomputed we at most need two bloom filters - available and used. This is easier than searching thru 65536 UUID's manually. In the event that we need to renew the pool of UUID's we can swap out the UUID's with a precomputed bloom filter. Meaning that there can be 100's of precomputed bloom filters and the associated 1MB of data to sent to the client. So this is work that can occur during idle times and during bootup of the application.

7 - Realized that I need to ensure that the header isn't modified so that an incorrect command isn't sent en route by using a throwaway public/private key pair.

8 - Researching Curve25519 if it's broken or not in terms of Quantum protection. It's secure enough since it's encased within XChachapoly1305 symmetric encryption. In the event

9 - Realizing that XChachapoly1305 is 256 bits of encryption and provides authenticated data of 12 bytes. So public/private key encryption isn't needed....

10 - Reading thru Standard Notes encryption lib to understand the tag size and modifying header to ensure data is KB aligned.
What is associated data that I keep seeing?
Taking a quick dip into https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-xchacha-01
Still don't understand what associated data is but I was able to verify that the tag is 16 bytes.

11 - Happy accident stumbled upon stable lib implementation of the encryption code which means it's possible to move off of Standard Notes crypto library meaning I'm not then tied to the AGPL requirements. If I want to.

12 - Realized that 1 KB header is extremely big for the protocol and I'm left with more than 800 bytes of empty space. Inspecting packet sizes and overhead of TLS. So it can hold approx 1K to 1.5K bytes. I'll keep the header to 256 bytes header which gives us a massive 136 bytes of empty space for us as payload.

13 - Alright had a massive 13 bytes after adding in the necessary control bytes that be configured or interpreted close to 254 ways!!!

I realized data acknowledgment is still a requirement which I'll address in a future update.

14 - Turns out with the current file network transmission concept 2SSL can support transferring files of up to 4.5 Petabytes!!!! WOWOWOWOWOWOW

15 - Discovered issue with nouce reuse within the implementation spec. Figuring out how to fix that. Since I've shrunk from 1KiB and it's now 256 bytes adding a bit extra for nounce is fine. I just need to figure out the layer to add that. Alright just fixed that. The header is now an odd. Alright figured out that I really fucked up the numbering on the header... O_o That's an issue for later today or tomorrow.

Day 28 TLS and odd jobs

What I did: A bunch of odd jobs

Time spent: 67.167


I had issues with CertBot so I discovered and used ZeroSSL. I had to first host the subdomain directly on SapphirePack to allow the certificate issue to work. Then after the ACME challenge was generated and passed I generated the certificate and then setup DDNS.

I tested it and it worked out!!! This is very big news because it saves me close to 500 or more dollars and also allows some special network setup which allows the server to act as a local backup, git server and the production server all in one (remember we're on a very tight budget here).

I'm also dealing with some issues with my provider which is frustrating but I'm remaining kind and considerate. :)

I also built a domain and reliance map so that I can see where the weakpoints in my architecture is along with how things are organized.

Finally I learned how git stash worked and began developing an organizational health bot. This bot provides a simple 'Yes/No' answer to tell me if everything is working as expected or if there are issues. Having a single Green/Red thing means that issues such as TLS, DNS and other messups are caught within the hour instead of days. Hopefully in most cases the bot can automate the solution or at least provide the printed out steps to fix the issue just like Phoenix framework does.

Day 27 Security Upgrade and Tentative Step for solving dynamic DNS

What I did: Ensured site sapphirepack.org only ran using latest 256 bit encryption and restricted to using TLS1.3.

Time spent: 65.150


The biggest thing is ensuring a high level of security for the static content of the site. For the dynamic content, I'm not willing to do cloud since I believe that we should be able to self control the most important hubs of Gaya.

This will be processing some of the most important data, acting as relays and such. There are several benefits and drawbacks to this approach. The biggest drawback that I can see is cost and economy of scales. Breaking into our servers will likely be waaay easier than breaking into Amazon's servers or Googles.

However by going with a public cloud provider I loose the protection of the server, I can't protect the server and anyone can tap and spy on the information. Gaya requires trust between the operators and Sapphire Pack.

Furthermore there are other issues that we haven't even considered like routing, latency and such. However if the company gets that big we can reevaluate the core infrastructure. The infrastructure that I'm designing is designed to be easily moved and redeployed.

I also began test running the idea of having a subdomain point to a computer hosted within our network so that it'll be more cost effective to run since renting a bare metal server on a cloud provider is very expensive. Even going Digital Oceans route can get quite expensive, Heroku I looked at but there were other issues and you're nickled and dimed. Quite frankly we need to cut our hosting costs as much as possible. We don't currently have the budget to be spending 60-80$ a month on a server, much less the certificates and other requirements to ensure that it all properly nests under sapphirepack.org

With this concept I'll likely need to explore Let's Encrypt to protect the subdomain on a local computer but it should be cheaper and easier then provisioning a server remotely. I'll run it for the first while with very aggressive protection under limited remote device interaction to avoid certain classes of exploits (hopefully).

Day 26

What I did: Plan backwards compatibility and thought experiment

Total time: 64.383


Now that I've split the session and security into two separate layers the first thing is ensuring session negotiation is compatibility. I'm pulling off of TLS concept for key negotiation and seeing if I can do it in a single round trip (since TLS has already done a few trips (2 or so I believe)).

Day 25 Simplify 2SSL and review Quantum Proptocol

What I did: Read and simplify protocol along with reviewing how the quantum protocol works and planning tomorrow.

Total time: 64.350


Realized I had mixed session and operator togeather. Simplifying and worrying about deauthentication attacks and denial of service attacks this way. In a future protocol may have a ignore deauth so that if communication is occuring over a hostile environment that it'll stand a better chance to make the communication succeed by ignoring likely false unable to decrypt messages.

Day 24

What I did: Finished migrating 2SSL documentation specs

Total time: 63.750


This was a much bigger job than I expected because the documentation ended up having alot of extremely big holes that had to be dealt with right away along with issues in terms of terminology, storage, bootup, migration.

The biggest issue is the need to design storage that doesn't place restrictions on how a system is implemented yet provide enough universality between implementations that future installations can interact with archaic data and vice versa. In the case it's the second state, I designed a .future() and .isDisabled() requirement for all storage engines that do the following thins:

For .future() prevents the storage from being used and the second indicates if this has occurred. There isn't a way to roll back after it occurs. This is setup during the storage design so that if an application has said storage provider, that regardless of which version of the application starts that system it'll know if it shouldn't run or not.

I've also taken the NestJS concept of modules and designed the startup system to be a 5 or 6 stage start which is nice because each part can be plugged and chained using a factory pattern.

The biggest issues that I've found with the 2SSL documentation was that it ended up being morphed as my idea of what 2SSL should do was morphed.

It combined security, data transport and session management all into one confusing layer. I've tentatively broken it into three layers and completely stepping away from having it be a drop in invisible proxy between express sessions on both devices. I think the send/respond lockstep protocol is inherently rigid and while it must exist as a low level shouldn't leak into the higher level development.

Day 23

What I did: Planned next big step

Total time: 57.883


1) Plan Next Big step (30 day target)
Done the goal is simple yet also quite complicated because it involves determining the architecture and overall complexity/simplicity of the system. I've settled on a 2SSL terminator -> Session Unwrapper -> Worker -> Session Wrapper -> 2SSL Intiator
2) Plan each set of 5 hours chunks
Here is the current plan:

5 hour milestone:
Transcribe and bring spec docs up to date digitally
10 hour milestone:
Finalize Kyber and secure library setup + testing
15 hour milestone:
Implement 2SSL state 0 to 1 API and abstraction layer sudo API in rough code
20 hour milestone:
Implement authentication and key distribution
25 hour milestone:
Implement and plan session storage client side abstraction
30 hour milestone:
Implement and plan session storage server side
35 hour milestone:
Add contingencies for following:
1) Too many connections (this prevents a database from being overrun or being filled with junk data or DDOSing the database layer, discovering the flow of internal data route (for example how congested, which 2SSL terminator and 2SSL initiators are being used)
2) Connection corruption retrieval locally
40 hour milestone:
Implement DDOS protection
45 hour milestone:
Plan relay server concept

         ┌──────────────┐        ┌─────────────────────┐
         │              │        │                     │
  ──────►│ 2SSL Terminator       │ Session unwrapper   │
         │              ┌────────►                     ├─────────────┐
         │              │        │                     │             │
         └──────────────┘        └─────────────────────┘             │
                                                                │                       │◄───────────────
                                                                │                       │
                                                                │   Worker              ├───────────────►
                                                                │                       │
      ┌──────────────────────┐     ┌─────────────────────┐      │                       │
      │                      │     │                     │      └──────┬────────────────┘
      │                      │     │                     │             │
      │     2SSL Instigator  │     │  Session wrapper    │             │
◄─────┤                      │◄────┤                     │             │
      │                      │     │                     │  ◄──────────┘
      │                      │     │                     │
      └──────────────────────┘     └─────────────────────┘

The beauty of this system is that it scales well for single and multi data center concept which is really powerful and forward thinking. :P
50 hour milestone:
Implement database contingencies
55 hour milestone:
Clean code and update documentation
60 hour milestone:
Provide working demo from Sapphire Pack to Heroku App or any other cloud provider.

3) Updated block regarding security from 'Setup two layers of security with the public/private key, including a hardcoded expiry requirement and list of pubic and private keys (signed with a primary key to authenticate and verify that changes are authorized).' Indicate first version will have hardcoded keys that will expire March 2023.

Day 22 Consolidating Ideas Sent

What I did: Consolidated Ideas

Total time: 56.467


An important part of any projects is managing the ideas. You may not use them now, you may not use them ever, but keeping good track of those ideas are super important.

For example did you know that SSL and TLS are not the same thing? TLS is more secure and modern than SSL.

I wanted to ensure that Sapphirepack.org was running with the latest security standards for encrypted connections.

For the frontend I'm likely to go with Gastsby since it includes the right meta and markup tags to ensure a high level of security. Just like with Javascript it no longer makes sense to code that stuff by hand.

Day 21 Nx Search

What I did: Tiny Nx repo understanding

Total time: 56.167


Day 20 Short term security fix

What I did:

Total time: 56.150


Setup two layers of security with the public/private key, including a hardcoded expiry requirement and list of pubic and private keys (signed with a primary key to authenticate and verify that changes are authorized).

Looked at TweetNacl and realized I'm going to have to wrap it along with Base64 because of how it works. Exploring the places that I'm going to need to use the public/private crypto and designing the 'perfect' api so I understand what I need to build in terms of the public/private key crypto infastructure. (This will require 2 one off scripts that allow:

Day 19 Basics of Testing

What I did: Basic Testing

Total time: 54.700


I got it to successfully generate a shared session secret. However there's a minor problem I just realized, this library just creates a shared session secret but doesn't allow using a long lived public/private key pair to verify the ssl2client is connecting to the correct ssl2server.

The use case is to prevent catfishing or hijacking the domain. However I'm coming up against issues that's making me rethink parts of the protocol.

Day 18 Monorepo and Security

What I did: Setting up the mono repo + importing library

Total time: 54.200


I got a MR accepted https://github.com/antontutoveanu/crystals-kyber-javascript!!! This feels so good even though it's a tiny change.

The reason that I'm going with a monorepo is to avoid as much software security chain violations. We've seen bad code get injected via a 3rd party dependency. By moving the most critical third party dependency as shared libraries directly managed and controlled within the repo, we remove an important attack vector, while opening ourselves up to a different kind of attack vector (not getting the patches we need because automatic updates aren't pulled).

However given the importance of security and the fact that the crypto libraries we're using have been formally checked or will be in the near future, we surmise the improved control over key crypto aspects will more than makeup for the hassle of having to manually manage and update the components.

I looked at four tools:

Bazel - Pass. Designed to run with multiple languages and such, configuration looks complicated and does more than I need it to do.

Nx - Interested, going to keep looking at others. Like that it has a dependency graph, a way to explicitly show what depends on what, which encourages looser coupling and discarding of old code that isn't needed.

rushjs - Backed by Microsoft, like how it's fully deterministic. Seems to be missing the testing hook but that's not part of the build process??? Whereas Nx does and since testing is important, I want the tools to reflect that. So pass.

Lerna - Pass. Doesn't have testing support.

Alright choose Nx, now need to remember to in the future add in a utlity that runs and tells us if any of our locally maintained security libraries have an update. For now I'm going to create a subrepo that's docs/security/watchlist.md

Alright good news I got the module setup, now I've got to setup the ssl2client and ss2server projects and figure out how to get them to interact within a test.

Sub project now, I'm importing the key code and setting up a few simple tests to ensure everything works as expected.

Hmm O_o kinda but not what we need. At least we're able to import modules from one package to another and vice vera.

There isn't anyway that I can currently know to track which depedencies go with which project.

Other comments:

In response to comment, absolutely good to be back. I discovered an issue that I couldn't address within SN's current security framework and the nature of the web. TO BE CLEAR, SN is extremely secure. So much that I trust my most intimate secrets and passwords wholeheartedly to SN. The issue was a combination of paranoia and the way that extensions are loaded via the internet. Nothing that SN can do to address it, an inherent flaw with how the internet is designed. So at that point I needed to abandon the effort until a better solution was available. I also found that I was using just a plain text editor and 1 note per password so even I myself was reluctant to use the password manager, never a good sign.

Day 17

What I did: Began dissecting the Public/Private Key Quantm crypto
Total time: 53.067

Day 16 Public and Private Keys

What I did: Researched public and private key system for over the air and quantum resistant key crypto
Total time: 53.033

Looked at DHole again for NIST-Finalists for Post Quantum crypto.

Here is the current realizations:

1) Need to negotiate a secure tunnel using Public/Private Key system that can resist quantum attacks.

2) The session once derived can use XChacha20-poly1305 which even using Grovers Algorithm gives 128 bits of security which is 'enough' for current day.

3)In the future since 2/3 finalists algorithms have a non zero decryption probability AND since these files stored in the cloud will be copies of files stored on said device, we can just ask the device to reupload the file that it failed to decrypt. However as Dhole mentioned 2-120 to 2-170 is quite dismissible for real world security.

Figuring out how to use the library. Submitted patches for the code. Nothing impressive moved some code around to fix scoping issues when converting all code for the 768 bit version to use let instead of 'var' and ''.

Day 15 Business research

What I did: Continued price research and cost analysis
Total time: 51.717