I made this, fun to see my project on the front page!
The main question I tend to get from hacker-types is "what are the tech specs?". The demos run on a Mac with custom software I built for projection mapping/rendering/etc. (the hand tracking uses Apple's built-in framework, which is very good), a small consumer 4k laser projector, and a basic webcam for vision.
But really, this is primarily a research/design prototype - I think it's much more important to figure out what the user experience should ideally feel like, if the interactions are even plausible & desirable in the first place; and then work backwards to the technology.
I met @heliographe a few years ago in a HN Tokyo (Slack) meeting. Happy to see you are still designing awesome things after your time at Apple. I look forward to seeing how your project(s) develop. PS: as soon as I saw the kitchen I knew this was Japan, and then I saw the hair and though —“hmm, I feel like I have seen this person before”— and it was you!!
Could you do beamforming with this, or would that get too expensive/awkward? At the very least, having more of these guys in the house (if you can get the tech cheap, ofc) would lead to better coverage potentially?
The main issue is probably gonna be that the display sucks compared to actual displays :( but it looks cool though! and the whole VLM thing makes a lot of sense
This is an awesome concept. How do you see this working across several rooms (if you've thought this far)? This would be great for when I have ideas in the shower.
The appeal of the lightbulb form factor is that you can mount it in a portable base to move it around the house; or in a fixed socket for broader room coverage.
It's not mentioned in the piece, but for the latter I don't see any technical obstacle to having several of these in different rooms in the house, or a couple in the same room to augment coverage.
A lot of people have Alexa or Google devices in each of their rooms, this seems like a very logical step. Perhaps 2 generations from what is being currently released based on how things go. Maybe sooner!
Is a basic webcam really able to read text at that angle for the “what is this word” example?
I wasn’t sure if any of this was real and that example looked extremely fake to me because I had a hard time believing the device being able to read text from that distance.
In that demo, the lightbulb sits on the desk, less than a meter away from the page. What feels implausible about this setup? This is kind of thing you can do with your smartphone today.
So is the demo real or is it smoke and mirrors? Could you have pointed at other words and have it work?
(I think I overestimated how much of an angled shot it was the first time I saw it, on top of how easy or hard turning your finger into “actually capture this text”… but you’re totally right that in principle a camera at a non agressive angle can capture that text)
(I like this overall concept a lot! Cooking is the right example, voice interfacing’s 96% use case is basically “my hands are dirty but I must use computing”)
It doesn't feel implausible, it is implausible, because your idea would require either a camera with good optics pointing in a specific direction, which means it's no longer a set-and-forget lightbulb, or some kind of 360 view more-or-less 3D camera to cover all the angles, which probably isn't going to have the resolution to read angled text at that distance.
At the very least it's going to have a range limit on that feature.
And there's a projector in there too.
In something smaller than a smart phone.
It's a great concept, but not a very workable one. Kitchens get steamy, tables can be huge, and management will want to use the cameras for user surveillance.
Its sad that search engines have become useless. There was another UX researcher who did something very much like this and then later got recruited to Google. As I recall he was using the cameras in Wii Nunchucks because they could do SLAM easily. And yet I cannot find his research, sigh.
Right. Tiny projectors have been done before.
Samsung had a smartphone with a projector back in 2014, but nobody wanted it.
Some startup called Moviphone has announced a new one for 2027.[1]
The use cases have not been compelling and fighting ambient lighting with a small projector is always tough.
Possible use case: projecting menus, ads, and bills on restaurant tables. With a camera and enough automated layout to work around what's on the table.
It might sell to the people who buy refrigerators with displays on the door.
Could be useful as a cooking training aid, like cooking videos but present where you need them.
The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
It's a good thing that most inventors don't fall for the "something vaguely like this has been tried with inferior technology before, and it sucked" line of reasoning, or our world would look very different :)
Well, if you're a inventor, you look at the prior art to see if it's been done before. I hold six issued patents, two of which were very profitable. I had one patent rejected because I missed relevant prior art from 1948.
Then again, I feel like I know industrial designers who play with building on ideas that came before, reexamining whether those ideas might have a fresh form and a place in the world today.
Who iterate and hone until they have something worth existing in today’s world—often doing do more publicly than corporations’ R&D departments—then let the patent technicalities be the well-paid lawyers’ problem…
Imagine downvoting John effing Nagle. However my humble opinion is that sometimes revisiting, if not reinventing, the wheel is beneficial in light of accrued epistemic benefits (or for the fun of tinkering with a cutesy creation) . Not for patenting, of course. Besides, the bulb form-factor is not too different from cctv cam ceiling installations.
> The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
gonna hard disagree! you aren't going to have 360° projector in there with wasted light going nowhere. you're going to have 1-4 steerable projectors. the "facade" as a globe bulb means you can place it at any direction you like depending on where the cord needs to go. for ceiling as well you don't want it to have a fixed throw direction. bulb makes perfect sense.
Nice. This reminds me how I felt watching the first pre-iPhone multitouch demos.
One small quality-of-life improvement that could be added: have the software timestamp when you say "here" and then (using a short low-FPS video buffer?) analyze the corresponding frame of video. This would eliminate the pause when you're pointing at something, waiting for the computer to respond.
Alternately a low-latency speech analyzer could look for the word "here" and quickly snap a screenshot. Perhaps project a temporary dot on the surface, so the user knows they can put their hand down now?
Quite the impressive demo! There's huge potential here.
Quite nice. But this imbues the house with a constant presence of the machine. A jinn which is always ready to grant your wish, but never leaves you alone, ever watching and listening. The fact that the jinn takes orders spoken in normal human language amplifies the effect quite a bit. I would have rather not have to feel such presence in my bathroom. Or bedroom. I would still be glad to be able to operate "smart" switches, and even find items. But I would rather explicitly unlock my phone, start an app, and scan the bookshelf, until a book is highlighted.
OTOH this would be a great thing for non-private spaces, that are watched by security cameras 24/7 anyway. A school, a library, a clinic, an office, a workshop, a warehouse, a parking garage, maybe even a museum.
Of course, trust and security are going to be major considerations. Maybe not for everyone though: people voluntarily enable various voice assistants, and even pay to order Amazon Echo.
I love to see the human-centric vision! This reminds me of Lenovo and Microsoft's "table computer" experiments from ~15 years ago. I remember being really excited at the possibility of my coffee table also being a computer--shared experiences, board games (I think lenovo even sold dice and pawns that connected to it). I had visions of having it also connected to the TV/Projector for controlling media. That was even before I had my first "smartphone". It could have been really cool!
One of the many cancelled form-factors the various FAANGs were playing with was a projector-centric assistant that screwed into a standard lightbulb fitting. The idea was it would placed over your desk/table and project downwards onto the surface. I still think its a very cool concept, even though none of the experiments ever panned out well enough for a real product.
Sadly if it ever gets manufactured it will use cloud and become just a better version of Flock. Even cat in the photo looks with obvious displeasure realizing this. Concept itself is novel and amazing though.
I've got something similar to this using cheap ONVIF security cameras and kasa smart bulbs. I have an nvidia card process the capture and for the smart speaker I use just a microphone and speaker hooked up to a normal computer. The architecture is spread across many machines and it's nowhere close to this realtime. Also the video capture is fairly lousy because the lighting and resolution and coverage isn't great. The cameras can change the lighting to try to get a better shot. (well it's qwen with toolcalls)
I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.
But I didn't think of the projector ... there's ~$30 ones ... might be worth buying a few.
Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.
Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too. I'm talking like ~$100 total
still researching them ... some of these look like garbage ... i'll come back to this thread in a few days when i find something decent. I don't want high nits or resolution, just want to be able to send pixels and not have it be noisy and get a response in under 500ms.
Author here; I was definitely inspired by my visits to Dynamicland many years ago, which Folk Computer is also inspired by.
One big difference in direction is that Dynamicland + Folk Computer use projector+computer vision as substrate to try and reinvent/rethink the interactive/dynamic medium, and what end-user programming could be (which is very cool).
In that way I am much less ambitious; this exploration is more about how there is an entire class of existing interactions (the ones showed in the piece) that we perform today on screens, that could be much more interesting + collaborative if overlaid in the real world. And I am also terribly uninspired by the eyewear-based computing vision all major companies have pushed lately.
An interesting concept. I'm (probably) one of the few people that thought the Humane Pin had an interesting idea at its core, it was just executed poorly at the wrong time. A stationary version that leans into ambience solves a lot of the interaction and engineering issues.
I also like that it addresses my gripe with XR interfaces; you aren't actually sharing a digital object in space and it looks weird to everyone else.
I love it. I only skimmed through the post but if this project has/will have a "hey lightbulb, where did I put my keys/wallet/brain" function I'm buying/building it on the spot.
What does the lightbulb have to do with the projection? Obviously the lightbulb is not the camera... nor the projector (nothing clear is getting through a frosted white shell). Is the lightbulb just a prop and imagined form-factor?
I feel like the demos are disingenuous because they suggest that the lightbulb is responsible for the entire thing.
I realize this is not the makers' focus, but I find myself less enthused with the human-computer interaction side and more anxious about how we can ensure the human maintains true ownership and agency over the device and its software.
I mean, merge one of these with a modern anti-consumer "Smart TV", and we're uncomfortably close to 1984's telescreen.
I totally agree. It feels like we’ve crossed a line where tech is overtly hostile to users because everyone is chasing recurring revenue and data mining is the only viable option
I think screens are time tested and will have some enduring use for now, but I could see people liking this; I've wanted to try a projector screen out but there was a whole rabbit hole of choices that didn't seem to improve upon screens for me for now (and this takes things in a whole other direction with projection)
I think the mission statement here is on to something. Helmets, obtrusive physical appendages etc are unappealing to me. The idea of using extant space or surfaces however, is extremely appealing. And while it might suggest lower quality today, as with most things, it would almost certainly improve over iterations and time.
Holographic computing, I think, will be the future. At least for interfaces.
I don't know about these things. To me they just turn everything into a screen.
Sure, it's a little more hands free, and projecting around definitly is nicer than screens. But projecting has a lot of problems too. Shadows, projection surface, ambiant light, heat are difficult to handle, but to me the worst is that it's not persistant. You can't just project 24/7 your family planner on the wall.
I'd really rather have something that prints, or better draws.
You want a recipe => print it, don't beam it.
You want a map of kyoto => tiny robot walks around the page and draws it for you. And you can draw over it if you want.
And it doesn't have to be exclusive, you can do both beam and print. But I think protable printing / drawing has a lot of unexplored potential.
I don't need the family planner to be on the wall 24/7 though, it can be projected only when I'm at the room. Maybe even only when I'm close to the wall.
If I printed each recipe I followed... gush that would be a lot of paper, and also I bet that paper would get messy very quickly.
For a map, losing the ability to zoom, mark points etc seems like a big loss. Sure you can draw on it (how do you erase your mistakes?), but if everything you draw just ends up lost when you discard the paper, that's not so ideal.
The future of computing is certainly multimodal. What I see with beaming is being both hands-free and prep-free further enhances the spontaneity of life and helps fill in small gaps.
Printing otoh requires hands and prep. That's greater effort. You wouldn't print a recipe if you were just a little uncertain about step 3, you would ask for some kind of just in time modality instead. Voice might be fine for short info but sometimes you'd prefer to have it up and there's beam.
yeah, I don't disagree, though you could think that a robot with a heat-paper printer or pen could go a long way.
But more importantly I ultimately think that one of the unstated goals is to reduce screen time. And setups like these are only one voice command away from beaming youtube videos all around your house.
Most of the videos are shot in real life, during summer daylight (only exception are the 2 person around a table ones, which are shot at night in a room with regular ceiling lighting).
And the projector I used is only 1100 ISO lumens; fancier models go way higher.
If you project mostly white content, it's remarkably readable + visible. Outdoors in direct sunlight would be another story, of course.
The main question I tend to get from hacker-types is "what are the tech specs?". The demos run on a Mac with custom software I built for projection mapping/rendering/etc. (the hand tracking uses Apple's built-in framework, which is very good), a small consumer 4k laser projector, and a basic webcam for vision.
But really, this is primarily a research/design prototype - I think it's much more important to figure out what the user experience should ideally feel like, if the interactions are even plausible & desirable in the first place; and then work backwards to the technology.
The main issue is probably gonna be that the display sucks compared to actual displays :( but it looks cool though! and the whole VLM thing makes a lot of sense
How many watts does the projector use for how many lumens? I would be curious if these numbers plus size would be a fundamental limit to the tech.
It's not mentioned in the piece, but for the latter I don't see any technical obstacle to having several of these in different rooms in the house, or a couple in the same room to augment coverage.
I wasn’t sure if any of this was real and that example looked extremely fake to me because I had a hard time believing the device being able to read text from that distance.
(I think I overestimated how much of an angled shot it was the first time I saw it, on top of how easy or hard turning your finger into “actually capture this text”… but you’re totally right that in principle a camera at a non agressive angle can capture that text)
(I like this overall concept a lot! Cooking is the right example, voice interfacing’s 96% use case is basically “my hands are dirty but I must use computing”)
At the very least it's going to have a range limit on that feature.
And there's a projector in there too.
In something smaller than a smart phone.
It's a great concept, but not a very workable one. Kitchens get steamy, tables can be huge, and management will want to use the cameras for user surveillance.
http://johnnylee.net/projects/wii/
Possible use case: projecting menus, ads, and bills on restaurant tables. With a camera and enough automated layout to work around what's on the table.
It might sell to the people who buy refrigerators with displays on the door. Could be useful as a cooking training aid, like cooking videos but present where you need them.
The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
[1] https://moviphones.com/
Who iterate and hone until they have something worth existing in today’s world—often doing do more publicly than corporations’ R&D departments—then let the patent technicalities be the well-paid lawyers’ problem…
Nothing truly new under the sun
gonna hard disagree! you aren't going to have 360° projector in there with wasted light going nowhere. you're going to have 1-4 steerable projectors. the "facade" as a globe bulb means you can place it at any direction you like depending on where the cord needs to go. for ceiling as well you don't want it to have a fixed throw direction. bulb makes perfect sense.
One small quality-of-life improvement that could be added: have the software timestamp when you say "here" and then (using a short low-FPS video buffer?) analyze the corresponding frame of video. This would eliminate the pause when you're pointing at something, waiting for the computer to respond.
Alternately a low-latency speech analyzer could look for the word "here" and quickly snap a screenshot. Perhaps project a temporary dot on the surface, so the user knows they can put their hand down now?
Quite the impressive demo! There's huge potential here.
OTOH this would be a great thing for non-private spaces, that are watched by security cameras 24/7 anyway. A school, a library, a clinic, an office, a workshop, a warehouse, a parking garage, maybe even a museum.
Of course, trust and security are going to be major considerations. Maybe not for everyone though: people voluntarily enable various voice assistants, and even pay to order Amazon Echo.
I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.
But I didn't think of the projector ... there's ~$30 ones ... might be worth buying a few.
Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.
Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too. I'm talking like ~$100 total
“Commander Riker is in his quarters.”
One big difference in direction is that Dynamicland + Folk Computer use projector+computer vision as substrate to try and reinvent/rethink the interactive/dynamic medium, and what end-user programming could be (which is very cool).
In that way I am much less ambitious; this exploration is more about how there is an entire class of existing interactions (the ones showed in the piece) that we perform today on screens, that could be much more interesting + collaborative if overlaid in the real world. And I am also terribly uninspired by the eyewear-based computing vision all major companies have pushed lately.
However, the light bulb concept is limited to indoor usecases - which is not bad - that's where people live most of the time.
The broader goal should be to get rid of dependency on screens, technologies and information. That's when AI can stop enslaving us with HITL etc.
I feel like the demos are disingenuous because they suggest that the lightbulb is responsible for the entire thing.
I mean, merge one of these with a modern anti-consumer "Smart TV", and we're uncomfortably close to 1984's telescreen.
I think screens are time tested and will have some enduring use for now, but I could see people liking this; I've wanted to try a projector screen out but there was a whole rabbit hole of choices that didn't seem to improve upon screens for me for now (and this takes things in a whole other direction with projection)
Holographic computing, I think, will be the future. At least for interfaces.
https://en.wikipedia.org/wiki/Storage_heater
Sure, it's a little more hands free, and projecting around definitly is nicer than screens. But projecting has a lot of problems too. Shadows, projection surface, ambiant light, heat are difficult to handle, but to me the worst is that it's not persistant. You can't just project 24/7 your family planner on the wall.
I'd really rather have something that prints, or better draws.
You want a recipe => print it, don't beam it.
You want a map of kyoto => tiny robot walks around the page and draws it for you. And you can draw over it if you want.
And it doesn't have to be exclusive, you can do both beam and print. But I think protable printing / drawing has a lot of unexplored potential.
If I printed each recipe I followed... gush that would be a lot of paper, and also I bet that paper would get messy very quickly.
For a map, losing the ability to zoom, mark points etc seems like a big loss. Sure you can draw on it (how do you erase your mistakes?), but if everything you draw just ends up lost when you discard the paper, that's not so ideal.
I guess what disturbs me is that I’d rather have one screen in my pocket than tens of screens in my house.
Printing otoh requires hands and prep. That's greater effort. You wouldn't print a recipe if you were just a little uncertain about step 3, you would ask for some kind of just in time modality instead. Voice might be fine for short info but sometimes you'd prefer to have it up and there's beam.
But more importantly I ultimately think that one of the unstated goals is to reduce screen time. And setups like these are only one voice command away from beaming youtube videos all around your house.
Youtube videos - https://www.youtube.com/results?search_query=sixth+sense+tec...
Reminds me of the projection bracelet scam that Captain Disillusion debunked years ago https://youtu.be/Cw7g8ixbomU?t=75
Most of the videos are shot in real life, during summer daylight (only exception are the 2 person around a table ones, which are shot at night in a room with regular ceiling lighting).
And the projector I used is only 1100 ISO lumens; fancier models go way higher.
If you project mostly white content, it's remarkably readable + visible. Outdoors in direct sunlight would be another story, of course.