rw | io

A Startup Post-Mortem

Wed, 01 May 2019 00:00:00 +0000

For the last couple of years, I’ve been building up a software testing startup, Verrify. Unfortunately, I haven’t been able to push it across the finish the line, and it’s time to put this startup to bed.

Verrify’s key idea is that you have to experience errors to handle errors, yet there are entire classes of errors – especially network ones – where tooling doesn’t exist to do so. As a result, our best efforts can give us a beautiful, smooth happy paths full of unnoticed gaps in the guard rails.

What happens when a network request times out? Or returns 301? Or when an internal service goes down and you can complete the first half of an operation but not the second? Maybe, raise your hand if you have a Gogo Inflight WiFi at your office to experience how your code handles captive portals? Without tooling to test these scenarios, we can’t write code that handles them. We need to be able to crash our code into the guard rails.

That’s what Verrify set out to do. I still believe in this core idea – so where did I fail?

There were plenty of unexpected technological hurdles to overcome – such as Apple deprecating the macOS network kernel extension system I was using, or the undocumented 15MB RAM usage limit on iOS network extensions – but it wasn’t ever software holding me back. Instead, I believe I made two key mistakes that were both rather trope-ier.

For one, I didn’t start shipping straightaway. It’s surely one of the most commonly repeated pieces of advice – just start shipping, get something out there even if its bad, and iterate with user feedback. I knew this, but clearly hadn’t internalized it. I spent a lot of time building and rebuilding until I was spinning wheels in polished ruts. I knew things /could/ be better and kept aiming above ‘good enough’, never quite reaching there.

Second, I tried to go it alone. While I started fast out the gate, over time I became less and less productive. I now understand why VCs prefer you to have a cofounder – an extra set of hands, a great check on my aim, yes, but also simply a person to share the journey with. At least personally, long term remote work isn’t for me. I’m more effective working around/near/or with a team.

Either shipping from the beginning or having a partner along for the ride would have shifted Verrify’s course somewhere farther down the road, successful or otherwise. But, while I wasn’t able to achieve what I set out to, it’s not as if all was lost. I’ve become a far more capable programmer, with several years more experience in C, C++, Objective-C and Swift. My knowledge of networking is night and day what it was when I started. I have far greater understanding & respect for the inner workings of a business and what’s involved in running one. And I better know myself, my strengths and my weaknesses.

I’m excited to jump back into a team and go again from there.

'Write for Six Month Ago You'

Thu, 24 Aug 2017 00:00:00 +0000

I really enjoyed Adam Wathan’s take on that feeling of inferiority when [considering] publicly sharing your thoughts & perspective. From Episode 16 of the podcast Fits & Starts (ref):

The biggest piece of advice I could give for people who want to be public with opinions, but are afraid maybe their opinions are wrong or they’re going to get backlash, or whatever. I think we have a tendency to want to impress our peers. You want to put out opinions and ideas and content that make the people that you admire admire you. If I could write a blog post and have DHH be like, “This was really insightful, I learned a lot from this! Wow!” That’s like the ultimate, you know what I mean? But it’s hard to do that sort of thing because you’re constantly terrified. Instead what I think people should be focusing on is teaching the people that are coming up behind them. Instead of trying to impress your peers, just try to impress six month ago you. There’s a million other six month ago yous out there. It’s kind of unintuitive, but the things you should most likely be sharing or writing about are things that are not really even that interesting to you anymore because you feel like you’ve figured it out. Those are the things you have the best odds of being able to share in a way that you’re least likely to be wrong, because as soon as it’s uninteresting to you anymore then it’s not controversial to you anymore and you’re most like just right.

We need more people teaching what they learned yesterday. A lot of the time you’ll learn something and figure it out, and you sort of forget what it took to cross the bridge to get there. […]. What was that I was trying to learn and why was it hard for me to find this information? How can I create the bridge for someone else who was looking for what I was looking for?

On Humble Inquiry

Wed, 23 Aug 2017 00:00:00 +0000

I have a recurring struggle where I find myself talking down to others when sharing technical information or pushing for my opinion on a matter. A friend recommended reading the book Humble Inquiry - The Gentle Art of Asking Instead of Telling.

I found it to be a great read. The writing is clear, full of examples, and at 144 pages the signal to noise ratio is superb. The line that finally made things click for me was this:

It makes you feel demeaned when you realize that they think you have not already thought of this yourself.

It always kills me to see something I’ve struggled to figure out for ages suddenly put into such simple words. “Ugh, it’s so obvious!”

A few other quotes from the book & the notes I wrote down alongside them:

If the choice is between “you or me”, look for a way to explore “us”, your relationship itself. Ask an open question to get information you need, a question that is not answerable with just a yes or a no. Invite joint problem solving.

“Do you want to grab lunch today?” —> “I can’t today, how about another time?” This puts the other person down, and denies you the opportunity to learn what may be behind the request + doesn’t allow you to share what has you focused elsewhere. “Do you want to grab lunch today?” -> “Mmm, what’s on your mind?” or, “What’s the occasion?” You learn what may be behind the request, can share what’s on your plate, and you can find a joint solution. It’s also caring, considerate.

Access your ignorance and allow curiosity to lead your questions, lead you. “What does the VP of Administration do?” [list] “Does PR have to be part of this job? Maybe we can separate that out.” When you lead with inquiry, you are seeking information, you are curious, you’re not jumping to conclusions nor feeling frustrated with the process or apparent obviousness of an answer. It facilitates an open conversation and invites change. These questions show interest and respect, they stimulate more truth telling and collaboration.

Asking for examples is one of the most powerful ways to show curiosity and interest and concern. More importantly it clarifies general statements. An open, follow up question is incredibly effective when problem solving.

Don’t jump in with an answer until you understand what is being asked. Put another way, don’t assume that the question that was asked was the [right/wrong] question. I frequently don’t answer the question that was asked as I’ve guessed at what they’re seeking and answered that instead. Rather, ask what they’re seeking and go from there, e.g., “Can you tell me how to defragment this drive?” -> “Sure, what are you solving for?”

And finally:

Clear your mind at the beginning of the conversation, and maximize your listening as the conversation proceeds. In fact, the most important diagnostic that the other person will use to decide whether or not you are interested is not only what you ask but also how well you hear the response. Your attitude and motive will then reveal themselves in your further questions and responses as the conversation proceeds. Humble Inquiry does not influence either the content of what the other person has to say, nor the form in which it is said. What I have found most important is to ask myself what my motives are before I ask a confrontational question.

On macOS Kernel/Kext Logging

Sat, 10 Jun 2017 00:00:00 +0000

I spent the better part of the last two days trying to figure out why I was seeing random / missing / corrupt log messages when trying to log from a kext on macOS Sierra. Sometimes the message was in the Console. Sometimes not. If I built a new copy of my kext and unloaded/reloaded, then the Console would just show random gibberish when logging.

The result of my suffering is that this specific combination works reliably:

#include < os/log.h >
* … *
os_log(OS_LOG_DEFAULT, “Your message with optional modifiers.\n”);

Then, from terminal:

log show –predicate ‘senderImagePath CONTAINS “MyKextName”’

(MyKextName is $(ProductName) in Xcode. not a com.whatever.thing. It is also case sensitive) That log command will show only Default messages and Fault messages. If you want to see Debug messages from ` os_log_debug( .. ) ` or Info messages from ` os_log_info( .. ) ` then include –debug and –info. But, these two types are not reliable in my experience.. Sometimes they log successfully, sometimes they don’t. I don’t see that issue with Default & Fault types.

Okay, so why log show, and not the Console.app or log stream? In particular, log stream cannot be used. stream cannot open the kernel process ID 0 for some reason. (I have SIP off, have sudo’d, and am running a development kernel, too), log stream will print some kernel messages, namely those generated by the OS itself. But stream will not show messages sent by my kext, at least. You can see some kind of error occur if you try to use ` log stream —process 0 ` or ` log stream —process kernel ` . Both error “PID not found for 0|kernel”.

I highly recommend playing around with Consolation if you have any trouble with the predicates - https://eclecticlight.co/tag/consolation/ .

Things I still don’t understand:

Why did loading a new build of my kext corrupt printf and/or IOLog output in Console?
Why, after a reboot, did the first load of my kext not printf anything in Console?
Are any of: running a development or debug kernel, setting nvram boot-args, disabling SIP, or creating a custom OS log identifier with os_log_create important or relevant? Not sure, but they don’t appear to be tied to my working combination above.
I read a lot about how tiny the in-memory log space is and how it gets overwritten if there is too much log printing. Is this relevant? I don’t think so, for my bugs here. I couldn’t print a single string reliably when no other activity was visible in the Console.
Should you run ` log config –mode ‘level:debug’ ` ? What does ` log config –status ` printing, for me, ` System mode = DEBUG STREAM_LIVE PRIVATE_DATA DEVELOPMENT ` mean?
Does checking Action -> Include Info Messages and/or Action -> Include Debug Messages change the apparent randomness in whether Console.app prints things? It didn’t for me..

Possibly helpful links to learn more about the system:

https://eclecticlight.co/2017/02/13/bugs-in-the-logs-sierras-new-logging-system-is-a-game-of-chance/
https://eclecticlight.co/2017/02/10/getting-more-out-of-sierras-logs/
https://eclecticlight.co/tag/consolation/
https://developer.apple.com/reference/os/logging
http://devstreaming.apple.com/videos/wwdc/2016/721wh2etddp4ghxhpcg/721/721_unified_logging_and_activity_tracing.pdf
https://forums.developer.apple.com/thread/66471
https://forums.developer.apple.com/thread/72834
https://forums.developer.apple.com/thread/66984
https://developer.apple.com/library/content/samplecode/Logging/Introduction/Intro.html

Last, for anyone searching…:

os_log
printf
IOLog
log stream
log show
Console
Consolation
Unified Logging
Activity Tracing
System_Library_Preferences_Logging/Subsystems
_System_Library_LogPreferences.plist missing kernel logs missing kext logs logs corrupt Console gibberish os x Sierra radar bug report 28948441 sysctl debug.kextlog debug.oslog

'Why Most People Get Stuck in Their Careers'

Tue, 21 Feb 2017 00:00:00 +0000

I enjoyed this piece by Scott Young. The distinction he draws between knowledge and meta-knowledge is a great framing device. ( link )

Doing well in your career requires two crucial factors: first, you need to be able to do your work well. This requires knowledge. If you’re a programmer, you need to master the languages you work with. If you’re an entrepreneur you need to know your market and how to serve them. If you’re a lawyer, you need to have a rich knowledge of the law.

However, this is only the first factor. The second part is that you need to have meta-knowledge.

Meta-knowledge is knowledge not on how to do your job, but knowledge about how your career works. That means you need to know which skills are the ones to invest in and which ones you should ignore. You need to know how to be able to demonstrate your skills to other people and the types of signals which carry weight in moving you ahead.

This second factor is often invisible and many people can go their entire careers without getting a very good picture of how people succeed beyond the station they find themselves in. One of our students, Chris L., didn’t even realize that he was missing it, “I was frustrated specifically because I thought I was doing a good job, and I see people who I don’t think are doing a good job and they’re getting ahead of me. I work hard, but nothing happens.”

'Heuristics Work Until They Don't'

Tue, 21 Feb 2017 00:00:00 +0000

From Scott Alexander at Slate Star Codex ( link ):

Anyone who learned the obvious lesson from Moneyball (“Pure math always defeats fallible human intuitions) would fail at the 2016 campaign, and anyone who learned the obvious lesson from the 2016 campaign (“Real experience and domain knowledge beat overeducated Big Data hotshots every time”) would fail at the 2003 baseball season.

The solution is: stop treating life as a series of moral parables. Once you get that, it all just becomes evidence – and then you wonder whether a single data point about Presidential campaigns necessarily generalizes to baseball or whatever.

At the same time, consider from Dave Gray:

“If you give people facts without a story, they will explain it within their existing belief system. The best way to promote a new or different belief is not with facts, but with a story.”

See also: this piece from Mike Caulfield on our Narrative Neediness ( link ) and these written 1995 remarks from Alan Kay ( link ).

In order to be completely enfranchised in the 21st century, it will be very important for children to get fluent in the three central forms of thinking that are now in use: “stories,” “logical arguments,” and “systems dynamics.” The question is “how?”

Why John Siracusa seems smart

Tue, 17 Jan 2017 00:00:00 +0000

In Reconcilable Differences #41, Merlin Mann & John Siracusa discussed being smart vs clever and why John always “seems so smart”. I found John’s descriptions of his thought processes enlightening, so I transcribed them for easy reference.

The transcription starts at 01:03:18 and cuts out some details for brevity. If you want to listen to it instead you can jump straight in here: https://overcast.fm/+EtBonnMDw/1:03:18

I think one of the things that I practice a lot, that I’ve been doing my whole life and practicing, is, I want to understand. There’s what you described as being clever, quick witted, able to associate things quickly, connect things in a way that makes some kind of sense and is often funny. But I always want to know how the thing works. We talk about this with little kids, like, “Why do you do the things you do? Why do you feel the way you’re feeling? Do you understand how you yourself are working?”

For example, some topic comes up and you free-associate a bunch of things about it, connecting it to the past conversation in a way that’s witty, its funny, people would call it clever, but you’re not preoccupied with figuring out, “Why was that funny? Why did I come up with that association? What am I trying to say with that comment?” You know, the death of humor, to dissect it and pull out all its organs and figure out how it works. Most people don’t like that, that’s why its one of the things that’s less common about me than other people, most people don’t have that urge to figure that out.

[…]

If you start to think about, “What were you even trying to say with that? Why is that clever? Is there some underlying point? Is there some actual connection or are you literally just free-associating like this neuron hits that one hits that one hits that one and words come out of your mouth?” But no one in that interaction cares about that.

For me, everything I’m taking in and putting out, I need to know why, I need to know what caused this to happen, what am I actually trying to say, what is the actual point. And that’s why almost every topic we talk about, you’re like, “Oh you already have all these opinions on things”.

I’m not going to let a podcast like that one with Hoffman enter my head without thinking through it and trying to figure what I think they mean and what I think about that and how it changes things and its not just going to be like, “Huh”, and let it stew in there. I don’t want it to be a black box. I need to know, every part of this whole process needs to be explicable.

And so, with everything I say and do, there should be some purpose. Even when I’m free-associating, I will retroactively try to figure out, “Where did that come from and why did it come out and what was I trying to say about it?”

Very often you think – I do as much free-associating as anybody else – but I’ll try to say, like, “Was that apt? Of all the things you could have pulled out of the giant bag of tricks that is your head, does that one make sense in context?”

For example, […], Because that association, that sort of like, connection that my brain made, I looked at it and said, “That connection makes sense. I understand, brain, why you made that connection.” It is not mysterious, its like, “Why do you do the things you do?” I know why that connection got made.

[…]

I always want to know why the things go together. If you go through any of my writing on my blog or even my OS X reviews they’re filled with pop culture references and links. I hope every single one of them is there for a reason. If I had to go back and explain before a jury, “Why did you link to this thing? What are you trying to say by linking this word to this thing?” I should have an explanation for every single one of those.

[…]

Where I’m getting is I think that’s why I might seem smart, sometimes. Not that I’m not smart, but part of seeming smart is when you’re saying things that when prodded about them can explain further. It’s like, “What, did you have all this prepared? Are you looking at a big set of notes about this?” No, the notes are all in my head. There is, every time you dig or prod or poke something to say, “Yeah, but what does that actually mean?”, I can keep going, and going down and down and down.

Its just a practice, its just like anything else. I am driven to do that, so I do do that. That’s why all of that is there. In the same way that some people are driven to make puns, or whatever. That comes off as seeming smart, because, getting back to the instinct before, when presented with a task that you think is going to involve some kind of thinking that you don’t think you’re ready to do, you’re afraid that, if poked, “Why did you try this?” your answer is “I don’t know, I don’t know why I just did that.” Whereas when you get that impression of me, if I was to try something and someone said, “Why did you try that?”, I’d have this big whole explanation of why I tried it. That makes me seem more prepared to tackle things. Even though, both of the things, we might have both tried the exact same thing and it might not have worked in either case. The fact that its explicable to me, in some ways it’s just bluster, or whatever, having some confidence. If you intuitively do the right thing, and I do the wrong thing but have a 10 paragraph explanation for it, who is actually smarter there? But I seem smarter, in scare quotes, because I have a reason, whereas you actually got the result but did it quote unquote intuitively.

Knowing why you do things and being able to explain things does not actually lead to better outcomes but it does make you seem smarter. Maybe. Or at least, more obnoxious, one or the other. No one likes a smart ass.

On Mobile vs Mac & PCs

Mon, 07 Nov 2016 00:00:00 +0000

Horace Dediu has a fantastic piece on the transition from PCs & Macs to mobile ( link ):

The big change in computing has not been a growing Mac vs. declining PC. It has been a huge surge in mobile device use vs. a decline in PC use overall.

This data is visible in many ways.

[…]

Indeed, because no usurper was allowed to emerge, PC/Windows never moved to a mobile evolution of computing. Microsoft’s platform future was lost because the antibodies which eat disruptions were left unchecked.

But Apple’s immune system was suppressed. It allowed a disruptor to emerge from within. Apple gave birth to its future by suppressing the reaction to that new seemingly parasitic organism. It took an immense willpower to allow this to happen.

But it takes us back to the question of what to do with the incumbent, the donor of DNA and resources. The parent that sacrificed for the child.

[…]

It cannot take on the role of being the future. That belongs to the touch screen devices. It will not morph into a touch device any more than a teen’s parent will become cool by putting on skinny jeans. What it will do is become better at what it is hired to do.

[…]

The Mac is what it is because it’s not alone. It’s part of a family. It is a parent. It strives to be better but will not take the future from its child.
The jobs that require a Mac or PC will continue to shrink & specialize. That’s inevitable, and okay. Bets on the future must keep that front of mind.

Benedict Evans, a few days ago ( link ):

A couple of years ago internet companies moved from having a mobile team and a mobile strategy to what they called ‘mobile first’. Instead of building a product and deciding how and if it would work on mobile, new things are build for mobile by default, and don’t necessarily make their way back to the desktop.

Now, though, I think we can see an evolution beyond ‘mobile first’. What happens if you just forget about the PC altogether? But also, what happens if you forget about featurephones? What happens if you presume all of the sophistication that a modern smartphone has and a PC does not, and if you also presume that, with 650m iPhones in use and 2.5bn smartphones in total, you can build a big company without thinking about the low end anymore?

‘Meditations on Moloch'

Mon, 07 Nov 2016 00:00:00 +0000

From Scott Alexander at Slate Star Codex ( link ):

There’s a passage in the Principia Discordia where Malaclypse complains to the Goddess about the evils of human society. “Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war.”

The Goddess answers: “What is the matter with that, if it’s what you want to do?”

Malaclypse: “But nobody wants it! Everybody hates it!”

Goddess: “Oh. Well, then stop.”

The implicit question is – if everyone hates the current system, who perpetuates it? And Ginsberg answers: “Moloch”. It’s powerful not because it’s correct – nobody literally thinks an ancient Carthaginian demon causes everything – but because thinking of the system as an agent throws into relief the degree to which the system isn’t an agent.

[…]

I described traps as when:

…in some competition optimizing for X, the opportunity arises to throw some other value under the bus for improved X. Those who take it prosper. Those who don’t take it die out. Eventually, everyone’s relative status is about the same as before, but everyone’s absolute status is worse than before. The process continues until all other values that can be traded off have been – in other words, until human ingenuity cannot possibly figure out a way to make things any worse.

That “the opportunity arises” phrase is looking pretty sinister. Technology is all about creating new opportunities.

Develop a new robot, and suddenly coffee plantations have “the opportunity” to automate their harvest and fire all the Ethiopian workers. Develop nuclear weapons, and suddenly countries are stuck in an arms race to have enough of them. Polluting the atmosphere to build products quicker wasn’t a problem before they invented the steam engine.

The limit of multipolar traps as technology approaches infinity is “very bad”.

Multipolar traps are currently restrained by physical limitations, excess resources, utility maximization, and coordination.

The piece is unbelievably long at 14,616 words, but was well worth the time. I took away numerous new perspectives on societal systems, outputs, approaches, [mis][aligned] incentives, end goals, and much much more.

Degrees of Freedom, Garden of Forking Paths — How to reach p < 0.05 every time

Mon, 07 Nov 2016 00:00:00 +0000

Another Scott Alexander piece, The Pyramid and the Garden ( link ):

Andrew Gelman writes about the garden of forking paths. The idea is: the scientific community accepts a discovery as meaningful if p < 0.05 - that is, if equally extreme data would only occur by coincidence 5% of the time or less. In other words, you need to win a lottery with a one-in-twenty chance if you want to get credit for discovering something absent any real effect to be discovered. But if a scientist forms their hypothesis after seeing their data, they might massage the precise wording of their hypothesis to better fit their data. If there are many different ways to frame the hypothesis, then they have many lottery tickets to choose from and a win is no longer so surprising. Gelman discusses a study claiming to find that women wear red or pink shirts during the most fertile part of their menstrual cycle, which sometimes involves red or pink coloration changes in primates. The study does detect the effect, p < 0.05. But there were a couple of different ways the researchers could have framed the problem. They could have looked at only red shirts. They could have looked at only pink shirts. They chose days 7-14 as most fertile. But they could also have chosen days 6-15 without really being wrong. They could have looked only at the unmarried women most likely to be trying to attract mates. A recent paper listed 34 different degrees of freedom that can be used in this kind of thing. Add up enough of them, and you have more than twenty tickets to the one-chance-in-twenty lottery and success is all but certain.