develwoutacause’s Twitter Archive—№ 138

Over the past couple months I've been working on a pet project. I just released 1.0.0 today and immediately archived the repository. 😕 I wanted to share a bit about the project and why it didn't really solve the problem I wanted to solve as well as I hoped it would...
Permalink On twitter.com 2020 Jun 20 Mood -1 🙁

…in reply to @develwoutacause
The project was called #BXPB - "Browser eXtension Protocol Buffers" (I know, I'm bad names). github.com/dgp1130/bxpb This was a library intended to solve a common problem I've encountered a few times while developing browser extensions.
Permalink On twitter.com 2020 Jun 20 Mood -4 🙁

…in reply to @develwoutacause
Browser extensions often run code in many different "contexts": an injected content script, a #DevTools panel, a background script, a popup box, etc. Each context has different privileges and APIs available. For example, only a content script can modify a web page's DOM.
Permalink On twitter.com 2020 Jun 20 Mood 0

…in reply to @develwoutacause
This means that cross-communication between the different contexts is critical, as many user stories require multiple contexts working in harmony. For example, a button in a popup that changes a page requires popup, background, and content scripts all working together cohesively
Permalink On twitter.com 2020 Jun 20 Mood +2 🙂

…in reply to @develwoutacause
Browsers themselves only provide some simple message passing APIs (similar to postMessage()) which can be difficult to work with, especially as a single context (such as a background script) scales to include more and more functionality over time.
Permalink On twitter.com 2020 Jun 20 Mood -1 🙁

…in reply to @develwoutacause
I wanted a way to essentially "export" a set of functions from one context and then call those functions from another context. This would make cross-communication nearly trivial in effort so extension development would be that much easier.
Permalink On twitter.com 2020 Jun 20 Mood 0

…in reply to @develwoutacause
I decided to use #ProtocolBuffers as a serializable data format to send across contexts. developers.google.com/protocol-buffers Protobufs already have the concept of a "service", so BXPB simply compiled a .proto file into a JavaScript client and service which communicate using messaging APIs.
Permalink On twitter.com ♻️ 1 Retweets 2020 Jun 20 Mood 0

…in reply to @develwoutacause
I was able to build this system as a couple NPM packages, but felt it was far too unwieldy to actually use. The getting started guide really exemplifies just how much effort it is to actually set up #BXPB with no path to making it any simpler. github.com/dgp1130/bxpb/wiki/Getting-Started
Permalink On twitter.com 2020 Jun 20 Mood -1 🙁

…in reply to @develwoutacause
So where exactly did this all go wrong? After a lot of thought on the matter, I was able to identify a few shortcomings of the design that doomed it to this fate.
Permalink On twitter.com 2020 Jun 20 Mood -3 🙁

…in reply to @develwoutacause
First (and most importantly), protobufs require their entire data model to be defined in... well... protobufs. You can't leverage existing JS/TS data types, and you must include **all** types which are transitively referenced by the top-level request/response types of a service.
Permalink On twitter.com 2020 Jun 20 Mood 0

…in reply to @develwoutacause
This is a huge amount of buy-in for a team to stomach before even starting #BXPB. This is only passable if you're organization is already heavily invested in protobufs (*cough* #Google *cough*) and/or you're already using them in your extension (such as #gRPC).
Permalink On twitter.com 2020 Jun 20 Mood +1 🙂

…in reply to @develwoutacause
Even if you are already using #gRPC, many types and data models used by an extension internally won't necessarily be passed to a backend. So this concept is highly likely to require additional protobuf definitions solely to support the extension itself.
Permalink On twitter.com 2020 Jun 20 Mood +2 🙂

…in reply to @develwoutacause
This problem isn't specific to #ProtocolBuffers, BTW. Any Interface Definition Language (IDL) would have the same problem.
Permalink On twitter.com 2020 Jun 20 Mood -4 🙁

…in reply to @develwoutacause
Second, existing protobuf tooling, particularly in the web space, is quite lacking right now. Improving #BXPB developer experience would mostly involve trying to improve #ProtocolBuffers developer experience as a whole. In particular:
Permalink On twitter.com 2020 Jun 20 Mood +4 🙂

…in reply to @develwoutacause
1. protoc (the protobuf compiler) cannot output #JavaScript using ES modules. You're forced into #CommonJS (or #Closure modules), which necessitates the use of a bundler for a browser extension which (surprisingly) would not need one otherwise.
Permalink On twitter.com ♻️ 3 Retweets ❤️ 1 Favorite 2020 Jun 20 Mood -1 🙁

…in reply to @develwoutacause
2. protoc cannot output #TypeScript types. Users are forced into community projects to generate typings for emitted #JavaScript. I was hoping to limit protobuf dependencies to #Google's officially supported projects only, but that simply isn't viable with #TypeScript.
Permalink On twitter.com ♻️ 2 Retweets 2020 Jun 20 Mood +3 🙂

…in reply to @develwoutacause
3. protoc is not distributed via #npm. There is no easy way to depend on the protoc binary in an NPM project. You need to install/build it separately, depend on community distributions, or use one from grpc-tools (official, but includes unrelated #gRPC stuff).
Permalink On twitter.com 2020 Jun 20 Mood 0

…in reply to @develwoutacause
4. Existing bundlers and build tools do not have officially supported plugins for protoc. Using generated code is often quite difficult, yet protobufs require this to get started. Officially supported integrations with #Rollup, #Webpack, etc. would make protos much more usable
Permalink On twitter.com 2020 Jun 20 Mood +3 🙂

…in reply to @develwoutacause
Because of these tooling limitations, convincing a team to use protobufs in a web project would be an uphill battle from the beginning. You'd need to expect a **lot** of value out of protobufs to put up with these challenges.
Permalink On twitter.com 2020 Jun 20 Mood -1 🙁

…in reply to @develwoutacause
Third, the main advantages of #ProtocolBuffers aren't all that helpful for the #BXPB use case. The main reasons to use protobufs today include: 1. Backwards and forwards compatibility. 2. Small wire format size. 3. Interoperability between languages. 4. Self-describing APIs.
Permalink On twitter.com 2020 Jun 20 Mood +4 🙂

…in reply to @develwoutacause
These features simply aren't very useful for browser extensions' internal use: 1. Extensions are released monolithically, so version skew and wire compatibility aren't an issue. 2. Communication stays on the same device, so wire format size isn't significant.
Permalink On twitter.com 2020 Jun 20 Mood +1 🙂

…in reply to @develwoutacause
3. Different extension contexts aren't likely to be written in different languages (unless you are mixing #TypeScript, #Elm, #Dart, and #GWT in really weird ways). 4. Better API docs is helpful, but not nearly as important when used internally in a monolithic application.
Permalink On twitter.com ♻️ 1 Retweets 2020 Jun 20 Mood +4 🙂

…in reply to @develwoutacause
The only meaningful features #BXPB gets out of protocol buffers is the serialization format and existing semantics (such as how a service is defined, how RPCs are defined, request/response messages, canonical errors, streaming RPCs, etc.)
Permalink On twitter.com 2020 Jun 20 Mood 0

…in reply to @develwoutacause
The end result of all these different problems with the current #BXPB design means that the barrier to entry is too high and the long term maintenance cost is likely as bad or worse than hand-rolling your own solution.
On twitter.com 2020 Jun 20 Mood -9 🙁

…in reply to @develwoutacause
I can't really envision a team which would choose #BXPB, use it over a couple years, and still be happy with the library then. I feel that anyone who would choose #BXPB, is likely making a mistake that they will regret later.
Permalink On twitter.com 2020 Jun 20 Mood -1 🙁

…in reply to @develwoutacause
So if #BXPB isn't the solution to my problem then what is? Since extensions are written in #JavaScript (and particularly #TypeScript), we should be able to leverage the existing type model to define a service. Introducing an IDL like #ProtocolBuffers hurts more than it helps.
Permalink On twitter.com ♻️ 1 Retweets ❤️ 1 Favorite 2020 Jun 20 Mood +1 🙂

…in reply to @develwoutacause
Based on this, I remembered @DasSurma's #Comlink library which solves a similar problem of communicating with web workers. It works with direct #JavaScript objects in a much simpler and lighter-weight manner. github.com/GoogleChromeLabs/comlink
Permalink On twitter.com ♻️ 2 Retweets ❤️ 1 Favorite 2020 Jun 20 Mood -1 🙁

…in reply to @develwoutacause
#Comlink doesn't *technically* support browser extensions at the moment, but there is an issue for it: github.com/GoogleChromeLabs/comlink/issues/438
Permalink On twitter.com ❤️ 1 Favorite 2020 Jun 20 Mood +2 🙂

…in reply to @develwoutacause
Ultimately, I think #Comlink is a much better solution to this problem than #BXPB. It doesn't require an IDL and it's a simple library with no compiler or other funky requirements.
Permalink On twitter.com 2020 Jun 20 Mood +2 🙂

…in reply to @develwoutacause
I wrapped up a 1.0.0 release of #BXPB just because I was so close to it already. Afterwards I archived the repo as I can't justify any additional development on it. It's still around for historical purposes if anyone cares to look at it more thoroughly.
Permalink On twitter.com 2020 Jun 20 Mood +2 🙂

…in reply to @develwoutacause
I still had a lot of fun with the project and learned a lot about #ProtocolBuffers, browser extensions, #TypeScript, #Lerna, #Rollup, and much more. I'm a little sad this project didn't work out, but I'm glad I did it. Now I just need to figure out my next project... 🤔
Permalink On twitter.com 2020 Jun 20 Mood +5 🙂