how i stopped worrying and loved the backdoor

A lie gets halfway around the world before the truth has a chance to get its pants on.
winston churchill

first of all i have to mention that netsec involvement was indirectly one of the first financial successes of theo de raadt (later mr.t for short) as the sale of 2500 cds through the EOUSA project (one for each us-ins office in the country) brought openbsd to profitable state and allowed mr.t to finance his living by means of the openbsd project.

but let us get back to our sheep (so to speak). as "the disclosure" from herr gregory perry mentioned the parts involved were ipsec(4)) and crypto(4)) framework and the "gigabit ethernet stack." but see? there is no such thing as "gigabit ethernet stack." moreover back then all the gigabit ethernet drivers came from freebsd. they were written almost exclusively by bill paul who worked at he himself does not always disclose where he gets the docs or other tech info for the driver development. drivers were ported to openbsd by jason@ (later mr.j). angelos@ (later mr.a) (who was contracted by netsec to work on the crypto framework in openbsd) was a post-grad student at at the time had contacts at columbia such as his friend and fellow countryman ji@ who worked there. ji@ wrote the ipsec stack initially (for bsd/os 2.0) in 1995. mr.a was porting it to openbsd. if memory serves me right it was during the summer of 2002 that a micro-hacking-session was held at for less than one week participating all the well known to us already mr.t and mr.j and mr.a with an addition of drahn@ and yours truly. primary goal was to hack on the OCF (crypto framework in openbsd). this does not affect crypto algorithms you'd say right? but why try to plant subtle and enormously complicated to develop side channels into math (encryption and hashing) when it's way easier to just make the surrounding framework misbehave and leak bits elsewhere? why not just semioccasionally send an ipsec(4)) packet with a plain text key appended to it? the receiver will drop it as broken (check your ipsec stats!) and the sniffer in the middle has the key! how would one do it? a little mbuf(9)) underflow combined with a little integer overflow. not that easy to spot if more than just one line of code is involved. but this is just a really crude example. leaking by just tiny bits over longer time period would be even more subtle.

here are just some observations i had made during ipsec hacking years later... some parts of ipsec code were to say at least strange looking. in some places tiny loops were used where normally one would use a function (such as memcpy(3)) or a bulk random data fetch instead of fetching byte by byte. one has to know that to generate 16 bytes of randomness by the random(4) driver (not the arc4 bit) it would take an md5 algorithm run over 4096 bytes of the entropy pool. of course to generate only one byte 15 bytes would have to be wasted. and thus fetching N bytes one-by-one instead of filling a chunk would introduce a measurable time delay. ain't these look like pieces of timing weaknesses introduced in ipsec processing in order to make encrypted data analysis easier? some code pieces created buffer underflows leaving uninitialised data or in other words leaking information as well. a common technique to hide changes was (and still is sometimes) to shuffle the code around the file or betweeen different files and directories making actual code review a nightmare. but to be just lots of those things had been since fixed (even by meself).

as the great ones teach us an essential part of any cryptographical system is the random numbers generator. your humble servant was involved in it too and right there in yer olde brooklyn. one breezy spring night i wrote the openbsd random(4) driver that was based on the linux driver written by theodore tso. and of course the output has never been statistically analysed since the day i wrote it. no doubt i ran some basic tests with help of mamasita (she's keen on math and blintzi). later the arc4 part was added by david maziers (dm@) who was also a friend of mr.a at the time and an openbsd developer. since then a number of vulnerabilities were discovered in the arc4 algorithm and subsequently the driver. most notably this potential key leak.

meanwhile in calgary... wasting no time netsec was secretly funnelling "security fixes" through mr.t that he was committing "stealth" into openbsd tree. (this i only knew years later when i was telling mr.t over a beer about the funny people i met on a west-coast trip (see later)). "stealth" means that purpose of the diffs was not disclosed in the commit messages or the private openbsd development forums except with a few "trusted" developers. it was a custom to hide important development in the openbsd project at that time due to a large netbsd-hate attitude (which also existed from the other side in form of openbsd-hate attitude; just check out this netbsd diff and an openbsd fix later; or a more recent "rewrite for clarity" commit that in fact changes functionality). which was a result of bulk updates of the openbsd sources from netbsd that we were doing back then due to the lack of own developers in many parts of the tree). in this massive code flow it was easy to sneak in a few lines here and there and make sure the "others" will not notice the importance of the change. of course this "stealth" attitude did not stop once openbsd got more developers and continued also in the ipsec areas (see for example). after all "security" was one of the main important keywords that were separating openbsd from netbsd back then. as we can see holding this funnel for netsec is putting mr.t on the payroll also.

actually it would be all too easy to spot the malicious code if it all be in the publicly-available sources. this leads us to believe that bits of the solution were in the hardware. unsurprisingly netsec was producing their own version of hifn(4) crypto accelerator. unfortunately hifn was refusing to disclose full docs for their their hifn7751 chip and that prevented the driver from being included in the openbsd base system. ( in the beginning the driver was called aeon since at that development was done on pre-netsec cards and the driver was renamed (see mv(1)) manually in the cvs repository files later on ). as a bit-chewing disasm-pervert i was asked to reverse engineer their "unlocking" program. that was some magic sequence (since then it's in the driver) that would initialise the hifn7751 after power-on and allow it to work. they had provided a sample program and challenged us. mr.t set up a machine for me in his house and i logged in remotely from my home in brooklyn to debug the c-code i devised from the disassembly of the unlocking proggy (see they did not even strip(1) it! ;). it was without any help from anybody else except for mr.t who was playing a role of my reset-monkey and yeah mamasita who was bitching at me for being late for dinner... and that worked. this was to show hifn that their "protection" is crap on the stack. the driver for the devices was written by mr.j who had access to public docs that lacked the "unlocking" sequence. this allowed netsec to start deploying their hifn(4)-based cards which by no doubt were a part of the side-channel scheme. about the same time at the bazaar show in nyc i was contacted by a representative of us-ins and a ukrainian millitary attache at un. both investigating my involvement with openbsd. a few months later i was offered an interview for a position at the fbi office for cyber-warfare in nyc who as well offered to fix my immigration status (or none thereof at the time ;) with a greencard. nonetheless for the lack of credibility from the future employer i refused to talk to them as for me deportation did not sound like an acceptable worst-case scenario.

soon enough due to professional contacts of mr.a the darpa grant for the openbsd was materialised. this was for two years work on various crypto technologies to be integrated in openbsd.

alot of the code resulting from the work sponsored by the grant still is in the repository except for parts that were done just for the noise and uncommitted later. of course no wander that darpa grant was spent primarily on mr.t and mr.j. i would expect mr.a was on benefit indirectly. three other developers on the payroll i suppose had to be there such it would not look completely obvious as a payment to mr.t and mr.j. initially mr.t offered me a position on it too but due to restrictions i could not be involved legally (as you can remember i had an expired immigrant status in the country of u.s. of a.). this was slightely disappointing as i had to spend money for coming all the way to philly for the meeting and as it seems for nothing. at least my trip to the following usenix anu-tech in monterey was payed by the moneys from the grant. at the time it only looked kinda funny to travel on the enemy capitalist government's budget ;) monterey by itself has not much of excitement but for the beach scenery and the cia agents for eastern-europe training camp. that would explain body search at the grayhound bus boarding (this was before the post-2001 scare) which ignored the knife and a whisky bottle i had in my pockets. before going to monterey and while exploring the beauty of san francisco i was contacted once by a us navy intelligence officer who seemingly unintentionally appeared next to me at the bar. later on my way back during a short stay in chicago also randomly appearing fbi agent. fellow was ordering food and beer for me and just like his navy pal gave me a warning to keep my mouth shut!


  • deraadt@ post
  • scott lowe's reply
  • jason@'s reply
  • more on the subject
  • -- 
        paranoic mickey       (my employers have changed but, the name has remained)
    all content copyright 2011 michael shalayeff
    back to ze mein