> So you're stuck debugging a system you don't control, through screenshots and copy-pasted logs on a Zoom call.
This is very real.
I work with a deployment that operates in this fashion. Although unfortunately, we can't maintain _any_ connection back to our servers. Pull or push, doesn't matter.
The goal right now is to build out tooling to export logs and telemetry data from an environment, such that a customer could trigger that export on our request, or (ideally) as part of the support ticketing process. Then our engineers can analyze async. This can be a ton of data though, so we're trying to figure out what to compress and how. We also have the challenge of figuring out how to scrub logs of any potentially sensitive information. Even IDs, file names, etc that only matter to customers.
I also used to work with on-premise installs of Kubernetes and their “security” postures prevented any in-bound access. It was a painful process of requesting access, getting on a zoom call and then controlling their screen via a Windows client and putty. It’s was beyond painful and frustrating. I tried to pitch using a tool like Twingate which doesn’t open any inbound ports, can be locked down very tight using SSO, 2fa, access control rules, and IP limiting but to no avail. They were stuck in their Windows based IT mentally.
> This can be a ton of data though, so we're trying to figure out what to compress and how. We also have the challenge of figuring out how to scrub logs of any potentially sensitive information.
This is fundamentally a data modeling problem. Currently computer telemetry data are just little bags of utf-8 bytes, or at best something like list<map<bytes, bytes>>. IMO this needs to change from the ground up. Logging libraries should emit structured data, conforming to a user supplied schema. Not some open-ended schema that tries to be everything to everyone. Then it's easy to solve both problems--each field is a typed column which can be compressed optimally, and marking a field as "safe" is something encoded in its type. So upon export, only the safe fields make it off the box, or out of the VPC, or whatever--note you can have a richer ACL structure than just "safe yes/no".
I applaud the industry for trying so hard for so long to make everything backwards compatible with the unstructured bytes base case, but I'm not sure that's ever really been the right north star.
Right, and that's when you do control the environment. Now imagine debugging that when it's your customer's infra, you have no access, and you're relying on them to copy-paste logs on a Zoom call.
IIUC this kind of thing is usually called “managed deployment.” Minio used to have a slick implementation of this, and I think databricks does as well. Usually it’s less “execute arbitrary commands on customer hosts,” and more “send metrics and logs to shared repository and send RPCs to customer deployment”
It's heavily inspired by Databricks' deployment model. And you're right that it's not "execute arbitrary commands". Commands are predefined functions in the deployed code that the developer defines upfront and customers can review.
The metrics/logs part is also core to Alien... telemetry flows back to the vendor's control plane so you actually have visibility into what's running.
It's not RCE. The commands are predefined RPCs written into the deployed code. Customers can review and approve them. Trust between the vendor and the customer is still required and Alien doesn't make it unnecessary.
I worked for a few years on an on-premise deployment of a system that was otherwise SaaS. Many enterprise customers simply won’t allow something like this - particularly big financials, aviation, healthcare etc.
Realistically, the game ends up being - see what you can get away with until someone notices. Given that, you might want to rename the product to something more boring than “Alien”.
In practice, unmanaged self-hosting is often less secure, because you end up with outdated versions, unpatched vulnerabilities, and no one responsible for keeping things healthy.
More and more enterprise CISOs are starting to understand this.
The model here is closer to what companies like Databricks already do inside highly regulated environments. It's not new... it's just becoming more structured and accessible to smaller vendors.
Both are real risks. But supply chain attacks exist whether you self-host or not... you're still running the vendor's code either way. The question is whether you also want that code to stay up to date and properly managed, or drift silently.
I agree that keeping things up to date is a good practice, and it would be nice if enterprise CISOs would get on board with that. One challenge we've seen is that other aspects of the business don't want things to be updated automatically, in the same way a fully-managed SaaS would be. This is especially true if the product sits in a revenue generation stream. We deal with "customer XYZ is going to update to version 23 next Tuesday at 6pm eastern" all the time.
This is true even with fully-managed SaaS though. There are always users who don't want the new UI, the changed workflow, the moved button. But the update mechanism isn't really the problem IMO, feature flags and gradual rollouts solve this much better than version pinning
"Written in Rust" seems to be a very popular thing to add.
My assumption is that people know it will get the thread more visibility?
A different take: https://www.cloudron.io/
This is very real.
I work with a deployment that operates in this fashion. Although unfortunately, we can't maintain _any_ connection back to our servers. Pull or push, doesn't matter.
The goal right now is to build out tooling to export logs and telemetry data from an environment, such that a customer could trigger that export on our request, or (ideally) as part of the support ticketing process. Then our engineers can analyze async. This can be a ton of data though, so we're trying to figure out what to compress and how. We also have the challenge of figuring out how to scrub logs of any potentially sensitive information. Even IDs, file names, etc that only matter to customers.
We're working on something for this! Stay tuned.
This is fundamentally a data modeling problem. Currently computer telemetry data are just little bags of utf-8 bytes, or at best something like list<map<bytes, bytes>>. IMO this needs to change from the ground up. Logging libraries should emit structured data, conforming to a user supplied schema. Not some open-ended schema that tries to be everything to everyone. Then it's easy to solve both problems--each field is a typed column which can be compressed optimally, and marking a field as "safe" is something encoded in its type. So upon export, only the safe fields make it off the box, or out of the VPC, or whatever--note you can have a richer ACL structure than just "safe yes/no".
I applaud the industry for trying so hard for so long to make everything backwards compatible with the unstructured bytes base case, but I'm not sure that's ever really been the right north star.
Same VPS, same config, but under sustained load you’ll see latency creep or throughput drift depending on the host / routing / neighbors.
Short tests almost never show it — only shows up after a few minutes.
The metrics/logs part is also core to Alien... telemetry flows back to the vendor's control plane so you actually have visibility into what's running.
Realistically, the game ends up being - see what you can get away with until someone notices. Given that, you might want to rename the product to something more boring than “Alien”.
More and more enterprise CISOs are starting to understand this.
The model here is closer to what companies like Databricks already do inside highly regulated environments. It's not new... it's just becoming more structured and accessible to smaller vendors.