技术博客与 newsletter feed · Nicholas Carlini · Writing

A fun experiment where I back-port all the improvements we've made to chess engines over the last 30 years and run it on a 1996 Pentium Pro to see what we could have done if only we knew more at the time.

An opinionated perspective on how to do important research that makes a difference (and sometimes win awards).

Are the harms that LLMs have caused, and will soon cause, worth the benefits they may bring? This article (a written version of a keynote talk I gave at CoLM) tries to explore this question.

A feature-complete gate-level microcoded Intel 4004 in 4004 bytes of C, capable of emulating the original Busicom calculator ROM for which the chip was originally designed.

A small hardware description language implemented as a DSL on Python, with a small 170 LoC 32-bit RISC CPU.

Future LLMs have the potential to cause significant harm due to their ruthless effiency. I'm worried this will happen, and discuss the ways in which it might.

I have very wide error bars on the potential future of large language models, and I think you should too. It's possible LLMs basically lead to AGI, and it's also possible they platteau.

My work on privacy-preserving machine learning is often cited by lawyers arguing for or against how generative AI models violate copyright. This maybe isn't the right work to be citing.

I have decided to leave Google, and will be joining Anthropic to continue my work on machine learning security

A one-year review of people's predictions on an AI-forecasting survey I made last year. Most people were over-confident in their predictions.

I wrote a (list of) regular expressions that will play a (not very good) chess game by running a 2-ply minimax search.

I let a language model write my bio. It went about as well as you might expect.

You should forecast the future of AI in this quiz, so that you can see just how right or wrong you are.

I don't think that AI models (by which I mean: large language models) are over-hyped. In this post I will list 50 ways I've used them.

Yesterday I was forwarded a bunch of messages that Prof. Ben Zhao (a computer science professor [a] A full professor with tenure, so I feel entirely within my rights to call him out here. at the University of Chicago) wrote about me on a public Discord server with 15,000 members, including this gem:

IEEE SP 2024 (one of the top computer security conferences) has, again, accepted an adversarial example defense paper that is broken with simple attacks. It contains claims that are mathematically impossible, does not follow recommended guidance on evaluating adversarial robustness, and its own figures present all the necessary evidence that the evaluation was conducted incorrectly.

A benchmark of ~100 tests for language models, collected from actual questions I've asked of language models in the last year.

How do I pick what research problems I want to solve? I get asked this question often, most recently in December at NeurIPS, and so on my flight back I decided to describe the only piece of my incredibly rudimentary system that's at all a process. I maintain a single file called ideas.txt, where I just append a new entry every time I think of something that might be an interesting topic for a paper. When it's time to pick my next project, I skim through the list, and pick whichever I think is mo

So let's suppose you had a 1980s Apple ProFile Hard Drive, and you wanted to recover the data.

Building a chess bot that queries GPT-3.5-turbo-instruct to play chess at the level of a skilled human player.

TODO

This program is a dependency-free implementation of GPT-2, including

I recently got back from attending USENIX Security 2022, and someone pointed out to me that it's been five years since I wrote Towards Evaluating the Robustness of Neural Networks (with my at-the-time advisor) and they asked if I had any thoughts on this paper. I didn't respond with that great an answer, but thought it was an interesting question, so I figured I'd write one down here instead. (In fact, I got the same question at IEEE SP earlier this year too, and didn't have a great answer then

A brief discussion about a tool I use to make rapid iteration in ML research possible.

A recent paper ('A Roadmap for Big Model') has copied a bunch of text from over a dozen prior papers. This is bad.

Abstract: Improving digital logic gates on Conway's game of life by allowing 8-bit logic gates instead of boolean logic gates.

This page contains a few checklists that help prevent embarrassing issues when releasing research papers online (e.g., via arXiv or a conference publication).

Abstract: An implementation of a minimal CPU on Conway's the Game of Life (an 'unlimited register machine'), and runs at ~10Hz.

Abstract: This post describes improvemnets made to my prior digital logic gate constructions (e.g., AND/OR/NOT) built on top of Conway's Game of Life, resulting in 100x faster simulations.

This year I entered in JS13K 2020, a game jam for JavaScript games in under 13KB (total size). I wrote a 3rd-person space shooter game, building on top of game engine I built last year for a doom clone.

InstaHide (a recent method that claims to give a way to train neural networks while preserving training data privacy) was just awarded the 2nd place Bell Labs Prize (an award for finding solutions to some of the greatest challenges facing the information and telecommunications industry.). This is a grave error.

For the third year in a row, I participated in JS13k 2021, where you're tasked with making a game in 13kB of JavaScript. Each year I enter participate I try to learn something new I didn't know how to do before. This year's motivation: I wanted to make a multiplayer game with some nontrivial networking aspects. So below you'll find a description of my experience building a MOBA in (barely) under 13kB of JavaScript.

I recently broke a defense to be published at CCS 2020, and this time I recorded my screen the entire time---all two hours of it. Typically when I break defenses, I'll write a short paper, stick it on arXiv, and then move on. Pedagogically, this isn't very useful. [a] (Don't you worry, I did that again this time, too.) So for this defense I thought I'd try something different.

Abstract: Using AND/OR/NOT gates built on top of Conway's Game of Life, this post walks through how to construct a actual circuits, for example a 7-segment display.

Abstract: This post walks through how to construct digital logic gates (AND/OR/NOT) on top of Conway's Game of Life, demonstrating its Turing completeness.

Abstract: We (again) broke a large collection of published defenses to adversarial examples. Here's how and why.

This year I entered in JS13K 2019, which asks people to develop games in under 13K of JavaScript. I entered a Doom Clone called ... Yet Another Doom Clone.

Late last year I decided it would be fun to build a 3D renderer in JavaScript. Recently it got into some sort of finished state and decided to put it here. This isn't so much of a tutorial on how to get there, but rather more of a here's a fun thing to do with nice pictures. But it was interesting to do. So here's that.

Abstract: A continuously-updating list of all 1000+ papers posted to arXiv about adversarial examples.

Abstract: This reading list provides an introduction to the field of adversarial examples for machine learning models.

Abstract: This document contains a collection of advice for performing adversarial example defense evaluations.