Bits are short notes that can’t or don’t need to be full posts. Follow them via RSS.

22 August 2025 Do not disrespect the fractal

Some people have a misconception that in software engineering, skill stops mattering for code quality from some level of seniority, and all of the value add shifts to architecture, high level design decisions, problem setting, or guiding others. And as long as you have staff/senior engineers design a system and oversee mid-level - and, for some work, junior - engineers, the output quality is the same as if you got senior folks to write everything instead.

What I believe, however, is that there’s not as much macro vs micro distinction as people want to imagine - software is fractal. Experts will make micro decisions, regular decisions and macro decisions, that all together combine into high quality software. Decisions at every level influence the quality and, often, influence levels above and below. You can outsource lower levels to non-experts - given time or budget constraints, you may have to - but you are not getting the same result.

You also can’t validate micro decisions from a macro vantage point. The process of making the micro decisions shapes your understanding of the problem; without having solved the problem from the ground up, you don’t have precise visibility into the higher levels. Quality engineering involves constantly shifting between the levels, validating the results and structure by looking at how much pressure propagates to neighboring layers and where things bend vs break.

This is why you shouldn’t replace engineers with LLMs even if you create a great plan and review the code. This is why you will not get software to improve if you outsource layers to non-experts. And this is also, I believe, why large teams routinely fail to make excellent software.

09 September 2024 Unlearning metrics and algorithms

The first somewhat social platform that I’ve used was LiveJournal; I used it around 2004-2010. Back then, we had posts and comments, but one of the notable features of the platform was the uni-directional friend relationships. The number of people who befriended you was somewhat of a status symbol, with a special term “тысячник” (a person with 1000+ reverse friend connections) used to denote Popular People.

That said, my recollection is that people mostly wrote what was fun or interesting for them to write about. Your friend feed contained a chronological display of whatever your friends posted - no ads, no algorithms.

At some point LiveJournal faded into irrelevance and Twitter was the hot new thing. Twitter started as a similarly simple platform - you had follow counts, and replies to tweets, but additional measures quickly entered the picture - these days, your tweet can be replied to, but the “engagement” indicators include retweets and likes. I’ve been meaning to write about this for a couple weeks now, and since then Twitter started also showing tweet view counts - yet another number that’s right in your face next to every tweet.

Once every tweet carries a set of visible engagement metrics next to it, it’s natural to start thinking about them. Am I reaching my audience? Was this a good tweet? How do I make my messaging more interesting?

Of course what also starts happening is that these - and other - engagement signals are used by the platform to form the content that people see. Long gone are the days of chronological timelines - Twitter still supports a linear view, but very aggressively selects an algorithmic timeline which is a setting that is stored per device/session. The algorithm takes visible cues, such as retweet/like numbers and progression, as well as less visible or documented cues - for example Twitter reportedly artificially reduces the reach of tweets that contain links, which makes it more difficult to share external content on Twitter.

This is a vicious cycle. Global factors like follow count are now not very meaningful - my account has 12K followers on Twitter (some of them bots), and yet some of my tweets get closer to 2K impressions1. By itself it shouldn’t matter as much but when it results in lower tweet signals, you start to wonder - am I doing something wrong? Should I not post this tweet because people aren’t going to engage with it? Should I post controversial or outrageous takes because that’s what generates buzz?

There’s really no rational reason for having these metrics drive the content, lacking an actual numeric (monetary) incentive - it shouldn’t make a difference whether 10 or 1000 people liked my tweet, and yet the number is right there, prominent, tantalizing, and the satisfaction of using the platform seems way too closely related to how high the numbers go.

What’s worse, for me at least this shifted the thinking about conversation. If my reply gets 100x less visibility than a tweet, should I even bother replying? Replies have smaller numbers so they’re less satisfactory and thus less valuable, or so the twisted thinking goes. Replying to Important People is numerically much more engaging than holding a profoundly interesting conversation - not a good outcome!

It’s because of all of this that I’m happy to see smaller, simpler, newer platforms.

Cohost doesn’t show you a single number as far as I can tell2. I don’t even know how many people follow me, and while I could probably find out if I tried hard enough - it doesn’t matter. What matters is the quality of the content I write, and the quality of the conversations in the comments.

Mastodon does show you a bunch of numbers, but the feed is chronological… and in fact, both the web client on mastodon.gamedev.place and Ivory, the iOS client I use, by default hide the boost and favorite numbers - which is a setting I’m happy to keep in its default, sane, position! As such the focus seems to be much more so on a discussion - and indeed, while it seems like the size of the user base is drastically smaller than Twitter’s, and my follower count is 10x smaller than it used to be, it feels like there’s a similar amount of interesting conversations that I actually want to read or engage in, at least in my field.

It’s still hard to not think about numbers that represent reach - in other networks like GitHub I still use the number of forks and stars to judge how popular a given repository is3. I still look at view counts on my YouTube videos to try to figure out what content I should publish and whether the whole video thing is worthwhile to begin with. That said, I’m trying to break away from caring about metrics and focus on content and discussion quality - numbers be damned.

31 May 2024 X is justifiably slow

I regularly hear or read statements like this: “X is slow but this is to be expected because it needs to do a lot of work”. It can be said about an application or a component in a larger system, and can refer to other resources that aren’t time. I often find these profoundly unhelpful as they depend much more on the speaker’s intuition and understanding of the problem, than X itself.

X1 may be slow because it’s using an inefficient algorithm. When a database query takes a while to return a large volume of results, it may be because there’s no index for that query - the query engine might be relatively well optimized, but the poor algorithmic complexity wins.

X may be slow because it’s using an inefficient implementation of an algorithm. It’s common to see 10x gaps or more between different implementations of the same idea, and without profiling the code it’s hard to know whether something unexpected is taking the majority of time.

X may be slow because it’s not taking advantage of the hardware. Modern hardware is incredibly fast, and many people don’t have a good intuition for just how fast computers they use day to day are. The performance deficit between a reasonable serial implementation and code that uses efficient SIMD and multithreading can be significant - it’s not particularly outlandish to see 100x delta on modern multi core chips with wide SIMD on computationally intensive problems - and replacing drastically cache inefficient algorithms with cache efficient ones can also yield dramatic speedups.

X may be slow because it’s actually doing the work, but doing the work isn’t necessary. Maybe X has no cache in front of it, or the cache hit rate is 10x worse than it should be, or maybe the cache retrieval itself is slow even though it doesn’t need to be.

X may not even use the right framing for a problem. C++ compilers are notoriously slow, but it’s not because the process of code compilation is fundamentally slow - it’s because every element of the stack often carries profound inefficiencies that can be corrected by reframing the problem (which may require significant changes to the formulation - maybe instead of C++ you need to compile a different language!).

And yet there are cases when “X is slow because it’s doing a lot of work” is actually probably right - when the problem has been well explored and can’t be reframed, when the implementation is thoroughly profiled and optimized, and especially when you can do some sort of speed of light calculation2, eg “on this system we can’t read memory faster than 50 GB/s, and yes we do need to read this much memory because we’ve already compressed the data to the extent feasible”.

It can be very difficult to tell the difference, which is why I get annoyed a little bit every time I hear this, because the odds that enough analysis has been done on the particular implementation of the particular solution on the specific hardware and the exact data that’s being processed before the statement is made are slim.

28 November 2022 All my editors are Vim, and yet none are

I get occasional questions about my development workflow - do I edit code in an IDE or do I use a simpler text editor, and if so, which one?

The answer to this question has been almost the same since 2009: I edit code in Vim. And yet for 3 years or so now the only command that starts Vim on my computers is git commit.

Let me explain. I’ve tried Vim in 2008 and quickly fell in love with the editor. I’ve used Far Manager (it’s a programmer-born-in-USSR thing, don’t ask) for a long time before that, so I was used to the workflow where the editor and the IDE were separate, but Vim was more powerful and efficient than editors I’ve used before. I quickly got into the Vim text navigation commands and macros, and they quickly became muscle memory - it’s now physically difficult for me to edit code without Vim-style motions.

It’s hard to do Vim’s motion commands justice in a short post, but for example in code like this:

		T& item = table[bucket];

… to change bucket to something else, I type f[lce1. This doesn’t require using the mouse, moving the cursor one position at a time with the arrow keys, or thinking - this is all completely automatic by now.

It’s also hard to know if this matters at all. Since I’ve used Vim editing for more than a decade, I can’t edit anything complicated without Vim, so in a sense I can’t reasonably compare my own typing efficiency with or without Vim.

That said, an area where Vim has consistently let me down is code navigation and project integration. I’ve tried a handful of projects that address this problem, and even wrote my own Vim plugin in Vimscript which I’ve happily used daily for quite some time.

Unfortunately, in unfamiliar or large codebases I found that none of the solutions could beat the quality of navigation that IDEs like Visual Studio give you; in small codebases I was missing the ease of switching between projects - all plugins I’ve tried or made for this required custom and slightly cumbersome setup that was fine for something you work on daily, but took too much time when you just wanted to make a quick fix in a random GitHub project.

Over the years, I’ve alternated between actual Vim and Vi emulation plugins. In fact, after learning Vim in 2008, I switched to Visual Studio with ViEmu plugin that did a reasonable job of emulating Vim key bindings and editing modes. Since then I’ve switched to VsVim for more modern versions of Visual Studio; when I’ve used Xcode I used XVim2; and when using Sublime Text, I’ve used Vintageous and then switched to NeoVintageous. Sometimes I would spend a year or two mostly using real Vim and then switch to an IDE for a couple years.

For the last few years, I’ve used Sublime Text as my primary editor where I spend 90% of the time writing code in, with occasional uses of Visual Studio (mostly when I’m in a tight edit-debug cycle which happens rarely these days, or when I’m in a very unfamiliar and complex codebase where builtin code navigation that Sublime Text provides is insufficient). Per above, in all the editors I use I have a Vim plugin installed.

The quality of Vim emulation, unfortunately, varies pretty significantly - Visual Studio’s VsVim is most excellent and covers my expectations from a Vim mode pretty much perfectly, whereas Sublime’s NeoVintageous still has significant gaps, mostly around macro usage, which gets in the way sometimes.

Curiously, I mostly use builtin code navigation in Sublime Text which is fairly rudimentary - instead of relying on something like clangd for precise queries, it uses an index built by building data for syntax highlighting using TextMate-style grammar, and allows to navigate using that index. This is somewhere between a full language-specific index built on top of AST with semantical analysis, that clang-based LSPs provide, and a full-text search like grep/qgrep.

This is reasonably close to ctags that Vim supports and so seems like something that Vim should be able to do with a plugin, and yet the fact that in Sublime this is available out of the box and works fairly well without me having to manage ctags files or find a ctags plugin makes it so that I didn’t feel the need to explore alternatives.

In fact, perhaps more important than the editor power for me is having the editor work out of the box these days. I change systems a decent amount and work on a lot of different projects at the same time, so doing substantial amount of setup per project or per system is not something I’d want to start doing even if it improves my workflow - unless the improvement is quite dramatic, which, between Sublime Text and occasional Visual Studio use, I didn’t feel the need for.

One popular editor that is very obviously absent from the list above is Visual Studio Code. It is being very actively developed to this day, has lots of easily available extensions (with the quality of extension ecosystem generally being substantially better compared to Vim or Sublime), so why not?

Every time I’ve tried VS Code in the past, the editor felt obviously sluggish to me - compared to the responsiveness around basic typing and more complex actions that I’ve come to expect after using Sublime Text and Vim a lot, typing in VS Code just always felt uncomfortable. This is something that I reevaluate every few years, and it’s something that slowly gets better with hardware and software improvements - I remember VS Code being completely unusable for me on a battery-powered Surface Pro 7, which is about the last time I’ve seriously tried it, so I’ll probably try it again!

One thing is for certain though: you can pry my Vim keybindings out of my cold dead hands.

23 November 2022 On greedy algorithms

I like greedy algorithms.

They are easy to implement, easy to reason about, and often pretty fast. A greedy algorithm makes a decision one by one and commits to it - no going back! Each decision is local and typically only evaluates a small set of choices. After the decision is made, you might need to update some tracking structures to reflect the change, and that’s it.

A lot of fairly complex problems have a simple, first-cut solution, that is greedy. You can then improve on that solution incrementally by tweaking the heuristic.

And yet it may be hard to understand the behavior of greedy algorithms at the scale of the entire problem. Each decision is easy to reason about in isolation, but it’s hard to know how the sequence of these decisions will affect the end result. Some locally optimal decisions, if made early, end up compromising a set of later decisions, locking down less optimal choices. Sometimes the heuristic must explicitly penalize locally optimal decisions, but it’s often not obvious how to do it, or whether a given choice is poor globally1.

I’ve spent a little bit of time in the last two days looking at the clusterization algorithm in meshoptimizer. It’s a greedy algorithm that fills the clusters with triangles one triangle at a time, and simultaneously tries to balance a lot of variables - how well packed the resulting triangle clusters (meshlets) are, how large they are, and how flat they are2. Any decision made early may end up isolating a few triangles - these are expensive to pick up later, as you’ll have to either have a lot of very “small” meshlets that are inefficient to render, or pack a lot of unrelated triangles into few meshlets, penalizing all forms of culling.

This is made harder by the fact that the incremental updates to connectivity structures done after each greedy step must be quick. Ideally I’d associate each triangle with the size of connected component and weigh triangles that are part of very small components higher - typical connectivity data structures for this are optimized for creating edges, not breaking edges, so I haven’t tried this yet.

The process of tweaking the heuristics is simultaneously fun and frustrating. It’s fun because, well, changes are easy to make! But even evaluating changes is difficult - it requires looking at statistics for a lot of different meshes and algorithm parameters because what improves one mesh might penalize another, and visual inspection is insufficient. And it’s often the case that intuition betrays you - a heuristic seems to make perfect sense, and indeed it does work great locally - except that when looking at the global algorithm behavior, it ends up flat or worse.

It’s the same mistake that the greedy algorithm makes, but it’s difficult to change the perspective and think about the impact of a local change globally. If you’re expecting tips & tricks for this, I have none :) I mostly wanted to share the experience.

15 November 2022 Write less to write more

I maintain a blog at https://zeux.io/ - which I started all the way back in 2007, at the beginning of my professional career. You would be forgiven to not know it exists - after all, in recent years I seem to average a couple posts a year.

I’d like to think that I’m a decent writer, at least as far as technical topics are concerned. People who’ve worked with me have seen long emails, Confluence docs and Perforce/Git commit messages - the way I write those is usually in one go, start to finish, and then either send immediately or skim once to correct mistakes. Why, then, do I not write blog posts that often these days?

Even though writing seems easy these days, writing blog posts still takes seemingly forever. I don’t track the time explicitly but it seems to take me about 6 hours to make a post. Writing is one part - at 3-4 thousand words and something like 60 WPM, it takes an hour just to type. But the bulk of the work goes into making sure my numbers are accurate, re-running / re-profiling programs, formatting code carefully for reading convenience, finding or generating diagrams and charts, and editing after the post has been written.

On the other side of the spectrum is Twitter. 240 characters per post is just too little, and thread-based posting is awkward, with limitations on code/layout. This makes it difficult to share meaningful thoughts, and trying to fit messages into a small series of tweets is a struggle. Cut too little, and you get a rambly thread with poor layout. Cut too much, and you remove nuance that results in a poor message at best, and starts a flame war in comments at worst. EDIT: And what if you made a mistake or forgot to add something?

Cohost seemingly makes these problems disappear, at least now while the platform is small1. No explicit size limit makes me feel like I can properly format my thoughts, the lack of explicit focus on virality and engagement makes me worry about mistakes less, and everything about the site somehow feels like a cozy cafe you stumble upon at the end of long walk and spend an hour in, just to come back again and again and again in the future.

I’m sure I’m going to write more blog posts at some point, and whether tweets or toots or something else survives is a convenient medium to share links to cohosts (is that a word?) or blog posts if nothing else, but, at least for now, Cohost is quickly becoming my medium of choice. Expect me to write about technology here, as well as about life in general. Most of what I share here will probably be written in the style that works best for me - written in one go, with minimal edits or preparation, with occasional image or code snippet that’s easy for me to grab.

One aspect I’m still struggling with is figuring out the reading aspect. I’ve long relied on RSS feeds, and that’s going to continue, but Twitter was also a source of extra reading material from people I followed. It’s not clear how this needs to shift now - do I follow people on Cohost? Does Mastodon replace that, and who do I follow there? Does Twitter survive and rebound? Time will tell.

13 November 2022 ARM: Pragmatism, not purity

Yet another train getting across France-Spain border, yet another post.

In the last few weeks before the sabbatical I was working on bringing up AArch64 support for native code generation in Luau. Before that my interaction with this architecture was limited to occasional glance at a disassembly emitted by the compiler. Coming from x86 world and having often heard how ARM is a much cleaner architecture than x86, I expected smooth sailing and clean, simple and unambiguous mapping between instruction bytes and assembly text.

Well, I came away from this exercise being somewhat disillusioned. I think what I expected was a pure and clean mapping with separation of concerns, but instead there’s many quirks of the ISA that I haven’t really expected1. It’s still less messy than x64, and I don’t even disagree with the choices made - but a lot of them seem to eschew purity and instead attack the problem from a pragmatic angle, stated as “what useful instructions can we fit into a 32-bit encoding space”.

Here are a few notes, in no particular order, of things that surprised me.

Overall, none of these are deal-breakers and many are pretty easy to deal with - that said, I expected the A64 assembler to be much simpler than our X64 assembler (for the subset of instructions we need), and instead it’s going to end up with roughly the same amount of complexity3. It now feels like the real benefit that A64 ISA has is the fact that the instructions are all 4 bytes in size, which makes it much easier to implement wide frontends4 - coincidentally this of course seems to be where all the irregularities come from, as you effectively need to devise an encoding that makes it possible to unambiguously encode a large set of instructions and as much data as possible in some of them. From the perspective of writing an assembler or a disassembler, however, I’m not sure the gap is that wide otherwise…

11 November 2022 Twitter financials are confusing

So the Twitter drama is ongoing and the fate of the platform is uncertain. I should say I generally like Twitter - or rather the community that has formed there - and I’m personally not looking forward to the migration and fragmentation of that community. Yes, I realize the irony of writing this on cohost.org but hey long messages > 280c messages.

One issue that is pretty important to the future of Twitter as a platform is financials. The company reported a loss of 270M in Q2 2022 - I definitely don’t understand the business enough to be certain about it, as in the last year and a half I learned that understanding financial statements is HARD. All of the below is likely to be completely wrong so don’t take it too seriously.

I remember looking at the data at the time of $20 => $8 Twitter Blue announcement, and comparing it with revenue numbers from Q2: $1076M ad revenue and $100M subscription revenue. Twitter Blue is $5 right now, so if you exclude Apple/Google share (which seems difficult to estimate from the earnings), $20 would have meant +$300M subscription revenue, putting the company in the green even before layoffs, right?

Well, Twitter doesn’t actually publish the subscriber counts in their earnings… but this data https://twitter.com/nickstatt/status/1589802376164040704 is interesting (this comes from insider reports and while in general that type of information is not trustworthy, that’s literally the only data point I could find, and insider reports to Casey Newton about Twitter have generally been true I think?).

According to the above, Twitter Blue actually only has 100K subscribers, which seems believable as Twitter Blue is a terrible value proposition right now (I’m not sure what the people who pay $5/mo are even paying for atm). And that reducing ads for these users by half costs $6/mo for each user, implying that the users of Twitter Blue bring Twitter ~12$/mo in terms of ad revenue.

I’m not sure if these numbers truly add up - if every Twitter user was worth $12/mo on ad revenue basis, then Twitter would have 9B ad revenue/quarter based on 250M mDAU, not 1B - but maybe Twitter Blue users are particularly engaged, which seems plausible even though we can verify none of this.

However, taking these at face value, this means that Twitter Blue quarterly revenue is just $1.5M right now (again, ignoring Apple/Google fees)1. This doesn’t square with the $100M/quarter in subscription revenue… but wait, the earnings report tells us, “This revenue line includes subscription revenue from the Twitter Developer Platform, Twitter Blue, and other subscription-related offerings”.

This must mean that the bulk of that revenue is actually large companies paying Twitter money for access to Twitter APIs. Twitter doesn’t disclose enterprise API fees, and their premium API access tier is $2500/month = $7500/quarter, so they’d need 10K developers to subscribe to get to that, which seems unlikely. That said, if enterprise fees are 10x the cost… at the very least this would mean that the “if you don’t pay for Twitter Blue you’re the product” is a fake narrative - you’re the product either way.

If 100K subscribers is right, then assuming the number of subscribers keeps growing, there’s probably some equilibrium point in terms of subscription revenue supplanting ad revenue. Which is to say, on average every Twitter user brings the company $1.4/month (based on ad revenue / mDAU), but in reality there’s still some sort of curve of time spent on the platform even among folks who use it daily. Financially it doesn’t make sense to get infrequent users to Blue with no ads, per report above, but if you actually grow the number of Blue converters to more meaningful numbers you can start to gradually offset the ad dependency.

As a Twitter user, I have to use alternative Twitter clients on mobile because Twitter’s ads (for me) are simultaneously insanely frequent (an ad tweet every 5 tweets including threaded replies is… bad) and poor quality. I would maybe pay $8 for Blue if it removed ads completely, but it’s not clear that the platform-wide conversion rate would support fully removing ads for Blue users, as that would require many more Blue subscribers to cross the ad revenue threshold. Maybe that’s the point of half-ads-frequency, as once positive movement in this direction is seen, lifting all ads can become more viable economically.

Overall I don’t really understand if Twitter has a plan from the user economics perspective. Being beholden to ads is pretty terrible for the state of the platform and for the users, but getting to a viable subscription plan could take a long time - the checkmark by itself feels pointless in the grand scheme of things.