I would advise not training your own model but instead use tools like langchain and chroma, in combination with a open model like gpt4all or falcon :).

So in general explore langchain!

I think you are absolutely correct for the interpretation of the photon count :)

I Googled it to see because I thought they maybe were using event cameras then but no, they use 10bit instead of classic 8bit but they are not litterally counting photons (which would not be useful). It’s interesting that it improved the precision and recall of their « object detection model ». Guess the image is of better quality then.

The link from 2 years ago is not particularly impressive: https://arxiv.org/abs/1406.2283 this is an equal valent paper I think from 2014

It’s an interesting discussion thanks!

I know that it can be done :). It’s my direct field of research (localization and mapping of autonomous robots with a focus on building 3D model from camera images e.g NeRF related methods )what i was trying to say is that you cannot have high safety using just cameras. But I think we agree there :)

I’ll be curious to know how they handle environment with a clear lack of depth information (highway roads), how they optimized the processing power (estimating depth is one thing but building a continuous 3D model is different), and the image blur when moving at high speed :). Sensor fusion between visual slam and LiDAR is not complex (since the LiDAR provide what you estimate with your neural occupancy grid anyway, what you get is a more accurate measurement) so on the technological side they don’t really gain much, mainly a gain for the cost.

My guess is that they probably still do a lot of feature detection (lines and stuff) in the background and a lot of what you experience when you drive is improvement in depth estimation and feature detection on rgb images? But maybe not I’ll be really interested to read about it more :). Do you have the research paper that the Tesla algo relies on?

Just to be clear, i have no doubt it works :). I have used similar system for mobile robots and I don’t see why it would not. But I’m also worried they it will lull people in a false sense of safety while the driver should stay alert.

The thing is working good enough most of the time is not enough. I haven’t driven a Tesla so I’m not speaking for their cars but I work in SLAM and while cameras are great for it, cameras on a fast car need to process fast and get good images. It’s a difficult requirement for camera only, so you will not be able to garante safety like other sensors would. In most scenarios, the situation is simple: e.g. a highway where you can track lines and cars and everything is predictable. The problem is the outliers when it’s suddenly not predictable: a lack of feature in crowded environments, a recognition pipeline that fails because the model detects something is not there or fail to detect something there… then you have no safeguards.

Camera only is not authorize in most logistic operation in factory, im not sure what changes for a car.

It’s ok to build a system that is good « most of the time » if you don’t advertise it as a fully autonomous system, so people stay focus.

If you use vscode, the foam extension is great!