Jim Fan

@DrJimFan599k followers

NVIDIA Director of Robotics & Distinguished Scientist. Co-Lead of GEAR lab. Solving Physical AGI, one motor at a time. Stanford Ph.D. OpenAI's 1st intern.

Open on X
Top outlier
136×
Best organic post
Peak views
16.8M
Single post
Bookmarks
156k
Across 100 organic posts
Last scan
Oct 16
Community cache
Posts 100 posts
#DatePost↓ OutlierViewsLikesRepostsBookmarks
1Dec 7, '23Grok just passed my sanity check136×16.8M29k3k1.2k
2Feb 16, '24Minecraft has been achieved internally Yes this is Sora's hallucination of Minecraft. It can't resist the urge to make the sky look less pixelated 😅57.2×7.1M7.8k4611.6k
3Feb 15, '24If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator 50×6.2M13k2.6k4.4k
4Mar 16, '23I asked GPT-4 to take over Twitter and outsmart @elonmusk. It comes up with "Operation TweetStorm"😮 and wants to publicly challenge Elon to a "Tweet-off showdown". Highlights: - G44×5.4M9.3k1.2k1.7k
5Mar 13, '23Million dollar idea: LLM keyboard. Every time I type on my phone and autocorrect makes a stupid mistake, it screams LLM. This is *literally* next word prediction. We should be typi37.2×4.6M4.6k332517
6Aug 9, '23The famed Stanford Smallville is officially open-source! 25 AI agents inhabit a digital Westworld, unaware that they are living in a simulation. They go to work, gossip, organize s32.5×4M9.4k2.2k6k
7May 26, '23What if we set GPT-4 free in Minecraft? ⛏️ I’m excited to announce Voyager, the first lifelong learning agent that plays Minecraft purely in-context. Voyager continuously improves 31.2×3.9M8.9k1.9k4.5k
8Mar 14, '23I don't give a damn about what is or isn't AGI. It doesn't matter. Below is GPT-4's performance on many standardized exams: BAR, LSAT, GRE, AP, etc. The truth is, GPT-4 can apply t25.7×3.2M6.4k1.2k1.1k
9Mar 22, '2310x engineer is a myth. 100x AI-powered engineer is more real than ever. As OpenAI winds down Codex, Microsoft announces GitHub Copilot X. I think it's almost as exciting as GPT-4 23.7×2.9M7.1k1.1k4.5k
10Oct 20, '23Can GPT-4 teach a robot hand to do pen spinning tricks better than you do? I'm excited to announce Eureka, an open-ended agent that designs reward functions for robot dexterity at 21.7×2.7M5.7k1.1k3k
11Nov 20, '23This is a master 4D chess move. WOW. 1. No new corporate structure. MSFT is literally one of the oldest for-profit tech companies out there, with a mature legal structure. Whether 19.7×2.4M6.5k7001.2k
12Jan 8, '25Allegedly shot in Shenzhen. Is this real? Can someone verify? I've seen this company posting very natural humanoid walking gaits a couple months ago. These days, it's hard to tell 19.7×2.4M4.7k5951.3k
13Nov 20, '23My team at NVIDIA is hiring. We 🩷 you all from OpenAI. Engineers, researchers, product team, alike. Email me at linxif@nvidia.com. DM is open too. NVIDIA has warm GPUs for you on 19.5×2.4M8.4k8031.5k
14Sep 24, '23Let's reverse engineer the phenomenal Tesla Optimus. No insider info, just my own analysis. Long read: 1. The smooth hand movements are almost certainly trained by imitation learni15.6×1.9M3k5321.5k
15Nov 24, '23In my decade spent on AI, I've never seen an algorithm that so many people fantasize about. Just from a name, no paper, no stats, no product. So let's reverse engineer the Q* fanta14.5×1.8M3.1k6373.4k
16May 12, '23AI Twitter is flooded with low-quality stuff recently. No, GPT is not “dethroned”. And thin wrapper apps are not “insane”. At all. I feel obligated to surface some quality posts I 13.9×1.7M7.5k9145.1k
17Jun 12, '23Today 6 years ago, "Attention is All You Need" went on Arxiv! Happy birthday Transformer! 🎂 Fun facts: - Transformer did not invent attention, but pushed it to the extreme. The fi13.9×1.7M4.2k9021.7k
18Apr 6, '23You think MidJourney's /describe is just a cool new tool? Think again. I believe hidden behind /describe is MidJourney's next-generation data flywheel. /describe guesses the prompt13.1×1.6M3.1k4381.3k
19Nov 23, '23It’s pretty obvious that synthetic data will provide the next trillion high-quality training tokens. I bet most serious LLM groups know this. The key question is how to SUSTAIN the12.7×1.6M2.5k2801.3k
20Feb 13, '23We’ve seen a gazillion startups using OpenAI APIs to do “co-pilot for X”. What’s next? Enter *physical* co-pilot! Here’s a compelling demo: you improvise by playing a “low resoluti12.5×1.5M6.7k1.2k1.4k
21Mar 10, '23*If* GPT-4 is multimodal, we can predict with reasonable confidence what GPT-4 *might* be capable of, given Microsoft’s prior work Kosmos-1: - Visual IQ test: yes, the ones that hu12.2×1.5M4.6k9501.5k
22Apr 12, '23AutoGPT is a prototype of the next frontier: "Agent Smith" AI that recursively clones itself. Achieved by (1) identifying *when* its context gets overwhelming and needs offloading;12.2×1.5M3.2k472745
23Nov 15, '23NVIDIA basically compressed 30 years of its corporate memory into 13B parameters. Our greatest creations add up to 24B tokens, including chip designs, internal codebases, and engin11.5×1.4M4.6k7862.4k
24Jan 20, '25We are living in a timeline where a non-US company is keeping the original mission of OpenAI alive - truly open, frontier research that empowers all. It makes no sense. The most en11.3×1.4M8.5k1.5k3.7k
25Mar 30, '23Chatbot UI: an MIT-licensed, community-driven clone of the ChatGPT UI. What most people don't realize is that you can pay *much less* to enjoy the same features as the official app11.2×1.4M3.4k4953.2k
26Jul 18, '23You'll soon see lots of "Llama just dethroned ChatGPT" or "OpenAI is so done" posts on Twitter. Before your timeline gets flooded, I'll share my notes: ▸ Llama-2 likely costs $20M+11.1×1.4M5.4k1.1k2.3k
27Jan 9, '23Here’s the recipe to make Siri/Alexa 10x better: 1. Whisper to convert speech to text. Best open-source speech model out there. 2. ChatGPT to generate smart home API calls and/or t11×1.4M5.1k9942.5k
28Apr 4, '23My guess is that MidJourney has been doing a massive-scale reinforcement learning from human feedback ("RLHF") - possibly the largest ever for text-to-image. When human users choos10.6×1.3M3.3k363723
29Apr 5, '23Reading @MetaAI's Segment-Anything, and I believe today is one of the "GPT-3 moments" in computer vision. It has learned the *general* concept of what an "object" is, even for unkn10.2×1.3M4.2k7161.5k
30Apr 14, '23We are looking at the future of VR, YouTube & Google Street View. This is zip-NeRF, a 3D neural rendering tech rapidly approaching the quality of a real, high-res drone flight 10.2×1.3M4k6711.3k
31Apr 2, '23HuggingGPT is the most interesting paper I read this week. It gets very close to the "Everything App" vision that I described a while ago. ChatGPT acts as a controller over the *AI9.9×1.2M5.3k8833.2k
32Apr 7, '23Why does generative AI struggle with hands? It is not a mystical Bermuda Triangle in the latent space. There're compelling reasons: 1. Data size (duh). Face pics are much more comm9.9×1.2M2.3k357709
33Sep 4, '23This is likely the most significant lawsuit in AI history - its outcome would have far-reaching impact on the whole industry. The arguments get fairly philosophical. Quote: "The pu9.8×1.2M2.6k5151k
34Jan 18, '23How to make ChatGPT 100x better at solving math, science, and engineering problems for real? Teach it to use the Wolfram language. ChatGPT: the best neural reasoning engine. Mathem9.4×1.2M4.5k6661.3k
35Apr 16, '23AutoGPT just exceeded PyTorch itself in GitHub stars (74k vs 65k). I see AutoGPT as a fun experiment, as the authors point out too. But nothing more. Prototypes are not meant to be9.4×1.2M3.8k501958
36Mar 21, '23I can finally discuss something extremely exciting publicly. Jensen just announced NVIDIA AI Foundations: - Foundation Model as a Service is coming to enterprise, customized for yo8.9×1.1M2.9k520946
37Sep 14, '23This is the way to unlock the next trillion high-quality tokens, currently frozen in textbook pixels that are not LLM-ready. Nougat: an open-source OCR model that accurately scans 8.8×1.1M3.8k7302.8k
38Mar 18, '24Today is the beginning of our moonshot to solve embodied AGI in the physical world. I’m so excited to announce Project GR00T, our new initiative to create a general-purpose foundat8.7×1.1M5.4k1.1k1.6k
39Aug 1, '23I'm waking up to the prospect that in my prime years, I'll see both mainstream superconducting and AGI. The former will propel the latter, and the latter will propel every scientif8.6×1.1M2.6k331338
40Feb 2, '23Music & sound effect industry has not fully understood the size of the storm about to hit. There’re not just one, or two, but FOUR audio models in the past week *alone* If 20228.5×1.1M4.2k8981.7k
41Dec 27, '22The AI explosion is warping our sense of time. Can you believe Stable Diffusion is only 4 months old, and ChatGPT <4 weeks old 🤯? If you blink, you miss a whole new industry. H8.2×1M4.1k9332.4k
42Feb 22, '24The first time I met Jensen was also the first time I met @elonmusk. I was interning at OpenAI that day and witnessed the moment Jensen handed Elon the first DGX. I slipped in my s8.2×1M4.3k268641
43Jul 13, '25I've been a bit quiet on X recently. The past year has been a transformational experience. Grok-4 and Kimi K2 are awesome, but the world of robotics is a wondrous wild west. It fee8.1×1M3.9k3131.4k
44Nov 18, '23Here's my prediction of what's next. The infinite energy of @sama & @gdb cannot be contained. They will re-build Rome from the ashes with even greater sense of urgency. OpenAI just8.1×1M2.5k248360
45May 13, '24I know your timeline is flooded now with word salads of "insane, HER, 10 features you missed, we're so back". Sit down. Chill. <gasp> Take a deep breath like Mark does in the demo 8.0×992k3.4k6152k
46Feb 16, '24I see some vocal objections: "Sora is not learning physics, it's just manipulating pixels in 2D". I respectfully disagree with this reductionist view. It's similar to saying "GPT-48.0×991k2.6k4251k
47Nov 18, '23I was OpenAI's first intern in 2016. I used to chat about the next learning paradigm with @ilyasut, engineering with @gdb, and scaling & safety with Dario. That summer reshaped my 8.0×987k3k175502
48Feb 16, '24Apparently some folks don't get "data-driven physics engine", so let me clarify. Sora is an end-to-end, diffusion transformer model. It inputs text/image and outputs video pixels d7.9×981k2.9k4131k
49Jan 7, '25Y'all expecting RTX 5090, cool specs and stuff. But do you fully internalize what Jensen said about graphics? That the new card uses neural nets to generate 90+% of the pixels for 7.8×959k6.6k5391.3k
50Mar 7, '23After ChatGPT, the future belongs to multimodal LLMs. What’s even better? Open-sourcing. Announcing Prismer, my team’s latest vision-language AI, empowered by domain-expert models 7.6×944k3.8k7571.8k
51Jun 28, '23Everyone should read the celebrated mathematician Terence Tao's blog on LLM. He predicts that AI will be a trustworthy co-author in mathematical research by 2026, when combined wit7.5×928k2.5k5991.4k
52May 3, '24We trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world. No fine-tuning. Just works. I’m excited to ann7.4×910k4.6k7721.7k
53Mar 14, '23GPT-4 is HERE. Most important bits you need to know: - Multimodal: API accepts images as inputs to generate captions & analyses. - GPT-4 scores 90th percentile on BAR exam!!! And 97.1×878k2.7k550630
54Apr 18, '24The upcoming Llama-3-400B+ will mark the watershed moment that the community gains open-weight access to a GPT-4-class model. It will change the calculus for many research efforts 7.1×873k2.4k404660
55Dec 6, '23This may be Apple's biggest move on open-source AI so far: MLX, a PyTorch-style NN framework optimized for Apple Silicon, e.g. laptops with M-series chips. The release did an excel7.0×871k3.4k5271.8k
56Jan 2, '25This is the most gut-wrenching blog I've read, because it's so real and so close to heart. The author is no longer with us. I'm in tears. AI is not supposed to be 200B weights of s7.0×861k3.6k3802.6k
57Feb 15, '23The Adam optimizer is at the heart of modern AI. Researchers have been trying to dethrone Adam for years. How about we ask a machine to do a better job? @GoogleAI uses evolution to6.8×837k3.5k5411.5k
58Sep 12, '23A neural network can smell like humans do for the first time!👃🏽 Digital smell is a modality that AI community has long ignored, but maybe one day useful for robot chef 👩🏽‍🍳? H6.8×837k3.8k7301.4k
59Aug 16, '23There're few who can deliver both great AI research and charismatic talks. OpenAI Chief Scientist @ilyasut is one of them. I watched Ilya's lecture at Simons Institute, where he de6.7×823k2.6k4082.7k
60Mar 27, '23Enough with LLMs - exciting things are happening in the world of atoms. This is Stanford ALOHA, a low-cost and agile robot platform. The whole system is open-source (!!): hardware 6.6×819k3.9k8801.3k
61Nov 25, '23Apparently people start to wear prosthetic fingers, so that surveillance images look like they're generated by Stable Diffusion 😅 The human race is overfitting to the quirks of ou6.5×807k3.5k459729
62Sep 12, '24OpenAI Strawberry (o1) is out! We are finally seeing the paradigm of inference-time scaling popularized and deployed in production. As Sutton said in the Bitter Lesson, there're on6.5×802k6.1k1.1k3.4k
63Mar 23, '23OpenAI just announced ChatGPT Plugins. If ChatGPT's debut was the "iPhone event", today is the "iOS App Store" event. 3 official plugins available now: - Web browser: adding Bing i6.5×802k3.3k5911.3k
64May 8I promise this will be the best 20 min you spend today! Robotics: Endgame, the sequel to my last year's Sequoia AI Ascent talk, "Physical Turing Test". I laid out the roadmap for s6.2×766k3.8k6004.5k
65Jan 11, '23Many people don’t understand how challenging Minecraft is for AI agents. Let me put it this way. AlphaGo solves a board game with only 1 task, countably many states, and full obser6.0×746k3k412602
66Aug 11, '23This is an ape ("Kanzi") playing Minecraft! A fascinating experiment on non-human biological neural networks 🙉 I've been teaching AI to play Minecraft for too long. There're so ma6.0×746k3.8k798950
67Jul 30, '23Kaiming He, inventor of ResNet, is leaving industry to join MIT faculty in 2024!! He’s one of the most impactful figures in deep learning. - Residual layer is a fundamental buildin5.9×731k3k327531
68May 12, '24OpenAI is expected to demo a real-time voice assistant tomorrow. What does it take to deliver an immersive, or even magical experience? Almost all voice AI go through 3 stages: 1. 5.9×728k3.2k5101.9k
69May 5, '24Congrats to @Tesla_Optimus team on another stellar update! The video gives us a peek at their human data collection farm, which I believe is Optimus' biggest lead. What does it tak5.7×709k2.9k391703
70Mar 28, '23GPT-4's vision API isn't public yet, but something better is here. Genmo: a creative & multimodal chatbot that not only takes image as input, but also generates and EDITs images an5.7×706k2.7k4362.1k
71Sep 20, '23I think DALL·E 3 is not just a stance against MidJourney. It's actually a sneak peak of the upcoming, epic battle of massively multimodal LLMs, against DeepMind Gemini. Quote: "DAL5.7×703k2.7k341586
72Feb 5, '24MidJourney hired an engineer from Apple Vision Pro to be "Head of Hardware". My best guess is that they are thinking about generating full synthetic worlds for AR/VR, because of th5.7×701k3.9k370885
73Apr 28, '23Transformers are here to stay for a while. Not because it’s the absolute best architecture, but because the staggering amount of resources lock us to the existing weights. Starting5.2×648k2.3k3891.2k
74Feb 8, '23Microsoft will let companies create their own ChatGPT. “BYOD”: Bring Your Own Data. Do you get the implication? Startups that are just thin wrappers around OpenAI API may finally g5.1×627k3.5k647876
75Feb 23, '24Career update: I am co-founding a new research group called "GEAR" at NVIDIA, with my long-time friend and collaborator Prof. @yukez. GEAR stands for Generalist Embodied Agent Rese5.0×618k3.7k4311.1k
76Aug 31, '23This is a neural network flying a drone at extremely high speed, beating human champions in FPV drone racing. - Reinforcement learning as a tool is so marvelously versatile. It's a4.8×599k3k600817
77Jun 30, '23Google is hosting the first "Machine Unlearning" challenge. Yes you heard it right - it's the art of forgetting, an emergent research field. GPT-4 lobotomy is a type of machine unl4.8×589k2.3k506745
78Dec 30, '24It gives me a lot of comfort knowing that we are the last generation without advanced robots everywhere. Our children will grow up as “robot natives”. They will have humanoids cook4.7×583k5.7k9951.8k
79Oct 30, '24Not every foundation model needs to be gigantic. We trained a 1.5M-parameter neural network to control the body of a humanoid robot. It takes a lot of subconscious processing for u4.6×569k3.1k4831.5k
80Jan 7, '25Introducing NVIDIA Cosmos, an open-source, open-weight Video World Model. It's trained on 20M hours of videos and weighs from 4B to 14B. Cosmos offers two flavors: diffusion (conti4.6×569k4.1k7021.8k
81Feb 4, '25We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed 4.6×568k2.9k445840
82Mar 16, '24We live in such strange times. Apple, a company famous for its secrecy, published a paper with staggering amount of details on their multimodal foundation model. Those who are supp4.5×559k4.2k7012.2k
83Mar 15, '23The wife trick that used to convince ChatGPT no longer works for GPT-4 😅. It's arguable what true human alignment should be here.😆4.5×558k2.6k251156
84Mar 18, '24Jensen Huang is the new Taylor Swift4.4×541k4k496423
85Oct 8, '24Hitchhiker's guide to rebranding: - Machine learning -> statistical mechanics - Loss function -> energy functional - Optimize the model -> minimize free energy - Trained model -> r4.0×500k4.7k6961.9k
86Jan 2, '23The launch of GPT-4 will be a predictably seismic event this year. But I can predict with high confidence what GPT-4 *cannot do*: It can’t cook spaghetti, play tennis, or build a l3.9×482k2.4k426202
87Apr 17, '24It took my brain a while to parse what's going on in this video. We are so obsessed with "human-level" robotics that we forget it is just an artificial ceiling. Why don't we make a3.9×481k2.7k368474
88Jan 24, '25Whether you like it or not, the future of AI will not be canned genies controlled by a "safety panel". The future of AI is democratization. Every internet rando will run not just o3.8×466k3k612768
89Apr 12, '24one day PhDs will animate every object around us with reinforcement learning to keep their thesis going3.8×464k3.3k330653
90Nov 30, '23One of the best tutorial-style repos since @karpathy's minGPT! GPT-Fast: a minimalistic, PyTorch-only decoding implementation loaded with best practices: int8/int4 quantization, sp3.3×408k2.5k3942.4k
91Jul 25, '25I'm observing a mini Moravec's paradox within robotics: gymnastics that are difficult for humans are much easier for robots than "unsexy" tasks like cooking, cleaning, and assembli3.2×399k2.5k527736
92Mar 18, '24Blackwell, the new beast in town. > DGX Grace-Blackwell GB200: exceeding 1 Exaflop compute in a single rack. > Put numbers in perspective: the first DGX that Jensen delivered to Op3.2×398k2.8k487540
93Dec 23, '24These are not CGI. Reinforcement learning is so back. When operating on strings, it gives us o3. When operating on physical motors, it gives us a perfect humanoid backflip and a ro2.9×357k2.7k414879
94Jan 27, '25An obvious, “we are so back” moment in the AI circle somehow turned into “it’s so over” in mainstream. > unbelievable shortsightedness > the power of o1 in the palm of every coder’2.9×353k3.3k409522
95Mar 15, '23Stop saying: AI will replace humans. Start saying: humans who know how to use AI at work will replace those who don’t.2.8×349k3k552162
96May 17, '24AI is math. GPU is metal. Sitting between math and metal is a programming language. Ideally, it should feel like Python but scale like CUDA. I find two newcomers in this middle lay2.7×331k2.6k4021.9k
97Nov 17, '23ChatGPT is now the new CEO.2.5×312k2.3k22130
98Jan 7, '25People ask me what's next. The GOAT points the way. Physical AI. Embodied Agents. Robotics. That's what's next.1.8×225k2.4k309637
99Dec 8, '22Why does ChatGPT work so well? Is it “just scaling up GPT-3” under the hood? In this 🧵, let’s discuss the “Instruct” paradigm, its deep technical insights, and a big implication: 1.3×02.3k4631.1k
100Nov 23, '22GPT3 is powerful but blind. The future of Foundation Models will be embodied agents that proactively take actions, endlessly explore the world, and continuously self-improve. What 1.2×02.4k492886

Scan another creator

@

Run this on any account.

XOutlier builds this table for every public profile you open on x.com, with live data and every post, not just the top 50.