Before we begin
This article will be about why I switched from Python to Go. This is from my experiences doing development in both over many years.
Like many articles of this type, some people might feel that their choices are being offended. That's really not the spirit of this article. This is more of a historical perspective on my part to show why I, and later the group I worked for, moved from Python to Go. If your a Python programmer, this article isn't meant to offend your choice.
But if you are a Python programmer interested in Go, this article is written for you. Switching was not an easy task, it required abandoning years of development in Python. I felt it was worth it and I believe we have reaped the benefits.
The Preamble
I've programmed in a lot of languages over the years. Most of these languages I remember almost nothing about.
Of course there was what I programmed in as a kid: Basic on a TSR80 and Commodore64, Hypercard and Think C on the old Macintosh OS and Pascal in some courses I took. I think I remember how to do the classic infinite hello loop in Basic and nothing about Pascal (other than remembering I liked C much better).
When I went to work for LucasArts/Film/ILM in 2000 I spent some time doing Windows Admin work, doing Visual Basic and C# for various utilities. As I became a Network Engineer, I moved to using PHP/Perl/Tcl/Expect(really Tcl) to automate work I really didn't like doing.
After a few years in a galaxy far far away, I left and joined Google. In the early days, I could program in basically anything I wanted. The predominant languages at the time were C++/Java/Python/Perl, but there was no infrastructure outside of some TCL and Perl scripts in my department.
I have never been a fan of C++ (it always seemed so bloated, which is funny coming from a guy programing in PHP at the time), Java looked like Factory().OtherFactory().SomeFactory(), and Python with its weird spacing instead of brackets wasn't attractive. So I started writing in PHP and doing some bad things so I wouldn't have to go get code reviews (there really weren't people in my department to do them anyway).
I wrote software that did things like global upgrades to all rack switches using PHP and TCL. I'm not sure I am proud of that.
As Google grew and we formed an official tooling group, Perl was on its way out and Python became king. If you wanted to interact with other systems via RPC, you had to use either C++/Java/Python. The tooling guys chose Python as their official language and I eventually conformed so that I could interact with these systems.
The Python Introduction
The thing about Google around 2005 was that we had Guido working there and other Python heavyweights at the time, so resources were everywhere.
As far as scripting languages go, Python didn't seem that bad. It had threads, multi-processing, a built-in web server, type reflection, ...
Other than its weird spacing where I'd use brackets, didn't seem like a bad language. Plus they gave me a free Python book.
Python didn't have real public/private variables/functions, it had this GIL thing, PYDOC wasn't that great for internal libraries and external documentation was only so so. It also had an inconsistent standard library around 2.4 . Things like method capitalization, camel casing, etc... were inconsistent. But this certainly wasn't a huge blocker.
The one thing I did like about PHP was it had great documentation with plenty of examples. Python certainly improved on their documentation over the years I used it, but back when I started, it wasn't stellar.
But all in all, I got used to it quickly.
Getting code right
One of the things you forget when you've programmed in dynamic language for a while is type safety and the love that a good compiler can show you.
The reason you forget about it is because of development speed. It is so much faster to start out in a language like Python or Perl than something like C or C++. C/C++ came with a lot of memory management you had to deal with and raw pointers that get people in trouble. Plus this is back in the days before nice little things like unique pointers and support for concurrency was built in. And let's not even talk about compile times and its effect on development speed.
But there is a downside to the speed you get early on with these languages: brittle software.
What years of writing large systems in Python taught me was that these systems had hidden bombs. Most systems have hidden bugs in them, but Python seemed to have a large number of these bugs.
A lot of the bugs that would be detected by a compiler in a type safe language wouldn't be caught in Python. So it might be a day, month or year later and BAM! Traceback!
There was a lot of work to try to limit these runtime errors by providing linters that could detect problems, but without those type hints it never got close to what a compiler can do.
We would try to compensate this with a lot of test code, but it often took years with very few changes introduced into the code base before you could stomp out all the bugs.
Memory performance counts
Then came performance issues.
Many of our systems started crashing due to out of memory issues. Ends up that Python was pretty bad about reclaiming memory. So eventually a lot of our critical services we had written started running out of memory.
This should have hinted that a new language would be needed, but instead we decided to hack in a new memory allocator called TCMalloc.
And at around 4x the cost for all our basic types, we certainly were using a lot more memory than we could get by going to C++.
However this brought us back from the precipice, but not without costs.
Using TCMalloc caused debugging problems and issues working in all environments (containers and desktop environments). The process of getting this to work with our internal build system was so confusing I had to write a paper on how to do it(based on the work of people smarter than me).
Thread performance counts
There was talk that because a lot of our software was IO bound, the GIL really wasn't much of problem. That didn't proved to be the case. Many of our services began becoming slower and slower.
We soon found ourselves trying to figure out how to split our services into smaller pieces to run as multiple jobs. In C++ this would not have been a problem and we could have concentrated on adding features. This restricted our mobility and forward movement by having us deal with peformance problems on a constant basis. Fanning out into multiple jobs leads to more complicated designs.
Multi-processing wasn't an option for us at the time, as it had some nasty SWIG interactions that randomly caused crashes. Plus I can't imagine that would have helped our memory issues with copying data across processes.
Like our TCMalloc approach, we would solve many of these issues by resorting to SWIG'd C++.
But it always felt like we were just plugging holes in a dam that was about to break with our fingers.
Binary size counts
So, Python didn't have a binary distribution method, but we had something similar for getting our code out to machines as a package.
This was problematic because the distribution packages were quite large. Importing libraries could have the effect of dragging in thousands of files that you didn't understand were included.
This was not a huge problem if you were comfortably in the US with 100G links and 1ms latency to your data centers. It was an entirely different matter if you were in a satellite office in Sydney connected via 1G to the US with 150ms.
When you were sitting in one of those offices waiting 5 minutes or more for a Python program on NFS to start, it feels a lot like the compiling comic from xkcd:
Some of these tiny programs would be hundreds of MB in size.
Version changes matter
You spent a year getting the bugs out of your good old Python 2.4 program and along comes the devtool guys wanting to change over to 2.6. Everything should just keep working, right?
I can remember tracking down thread locking problems in a program was working for a year or more only to find that an interpreter version change in the fleet was making the program crash.
It actually wasn't that the version was doing something wrong, it was that changes to how thread timing worked had uncovered a bug.
Now you might be saying, "hey, a compiler change would do that too". And you'd be right.
But what a compiler change wouldn't do is make my program stop working in production. If I compiled a new version, pushed it out and found it broken, I could always roll back a compiled binary.
But you don't always have that option when your interpreter changes, unless you also control the devtools. And devtooling was a little out of my control.
I'm sure there are ways to handle this with multiple interpreter installs or distributing an interpreter for each program to run, but that's really a lot of overhead you are adding.
Plus, I can only imagine that if I was at a smaller company, distributing a binary has got to be easier than all the setup required to install a Python/Java/Ruby/NodeJS program.
Introduction to Go
So somewhere around 2010, myself and few of the network tooling engineers are thinking that we've made a mistake. Our maintenance costs are really high, bugs are hard to stamp out, machines are getting more cores but the GIL isn't giving us more speed. Python was kinda like the car we'd driven 300k miles. Its wore out and we don't want to take it to the shop one more time.
Now I am contemplating if I should start programming in C++ or Java. Or I say that, but really Java never had a chance. Looking around internally, I hadn't ever seen a truly successful infrastructure project in Java and I didn't want a language that seemed to require an IDE to do my work.
C++ wasn't really exciting either. At the time there were like 8 different pointer types being used and the language, while not Perl, looked like everything and the kitchen sink is thrown in it. Splitting parts of the object definition and code between my .cc and .h files was not attractive (and I never liked having header files even in C). So I was dreading going down this path.
I had two colleagues in Sydney who had mentioned Go to me (they were sitting next to the Go team). Of course, I was thinking this was some other pie in the sky project that was going to burn out fairly quickly. Rumors had said it was slower than Python, and that tickled my "no" response.
But I was getting to the point of desperation when another colleague of mine in Mountain View, who was a fairly die-hard Python fan, started raving about Go. This was out of character for him.
The next thing I know, he had started to write a service in Go (against all his colleagues wishes). And he was doing this knowing that no one else knew how to program in Go, which was a support issue. He was a pretty "by the book" kind of fellow and with him feeling strong enough to do this made me interested.
The problem is, there were no learning materials really. This is pre-Go 1.0, so godoc and a few papers is about all you have. At this time there were probably fewer than fifty people outside the devs at Google who knew the language. So the only people who could review my code was the Go devs. Just slightly intimidating to send code to these guys. Let's just say it wasn't like sending it to any regular group of software engineers.
This made the learning curve quite steep. And the documentation wasn't very good. Or at least for someone like myself that learns by reading examples instead of the osmosis some people seem to have. I don't think any of the devs at the time had ever even heard of examples in documentation.
Stability was also a problem. You'd be working on your code for a week, everything would compile. You would come in on Monday, code wouldn't compile. That could be because things like the "time" library had been re-written. If your code was checked in you would have been automatically migrated. But still developed code was never safe.
However, none of that mattered. I was in love!
Up until this point, every language I used was just a tool to build something. The truth is, I didn't really care for languages, they were just necessary evils. I love building things: software, movies, lego blocks, whatever... I always thought of language love like loving a hammer, seemed kinda silly to me.
But for the first time, I had found a favorite hammer.
(To be continued in Part II)