Friday, January 25, 2019

Simple, stupid, effective tests in go


What

This is not about fancy TDD practices, what test frameworks you must use, or why [insert your favorite oldish (as in 2010s) jargon] cannot answer the exceeding amount of sophistication in our new software and [insert your favorite 2020s acronym] is the new shit.



This is about how we can overcome simple, yet annoying problems that come up in the process of writing code by writing tests.
I’m gonna talk about go here. Probably none of these can be applied to any other language/environment directly because of the differences in the language features, best practices, and/or conventions. However, I believe the analogy is a reusable one and therefore more general than just golang since I have caught myself thinking about the same practices in other situations.
Throughout this post I’ll sometimes refer to a hypothetical caching interface defined as:
in the package github.com/someone/cache. (No it does not actually exist [as of this writing])

1. Test struct “constructors”


The first thing that we write in a new .go file is often a struct. Suppose we’re going to implement the above interface on top of a redis backend. Let’s call the implementing struct RedisCacheService. Let’s start off by writing a test:
file: cache/redis/service_test.go
file: cache/redis/service.go
This is a very simple test. It may even look too simple since there is not even an assertion in the body. That’s because the compiler can do all the necessary work here by checking the function signature.
Writing this seemingly trivial test against the NewRedisCacheService(string, string) function not only allows you to advance src_test.go and src.go together (in that order) from early on in a full TDD manner, but also allows you to think about the usage of the “ctor” before actually writing it. Its easy to stay with an obviously problematic “ctor” interface as a result of justifying it in hindsight after you’ve written it down.
Also regarding the general testing philosophy, you may not be the actual user of NewRedisCacheService and therefore may have no way of finding out about breaking changes to this interface when you recompile.
This could also be library code and it could well be the case that you’re testing each functionality of the CacheService by manually initializing the RedisCacheService struct (as opposed to calling NewRedisCacheService) for example to mock some resource internal to RedisCacheService to which you don’t have access through the “ctor”.
The consistency of the ctor is easy to overlook when you’re writing tests, since it’s not an actual “functionality” of a defined interface, and it is obviously very destructive if an unwanted change is introduced.

2. Test interface implementations


Your implementation of the CacheService needs to be correct in terms of language semantics: The struct needs to define the methods specified by the interface.
This is also hard to keep track of as there might not always be explicit calls to every interface functions or pieces of code to “cast” up from structs to interfaces for the implementations of the functions to be checked. Also in the early developments of a new module, you would risk pushing semantically incorrect code to the upstream since there could be no usage of the new code yet.
Let’s keep things simple:
Add to file: cache/redis/service.go Again, there’s no need to assert anything. The compiler is doing all the work by checking the whether or not RedisCacheService implements cache.CacheService. Note how we kept it as minimal as possible by not calling the ctor and creating the object directly.

3. Test generated assets: mocks


When you specify the interfaces through which your program modules are supposed to talk to each other in the top level and only communicate through those interfaces, interesting stuff tends to happen. Here’s an example from digest:
A particularly pleasant thing is that you can generate all (well, except third parties) the mocks that you would need (you only communicate through these, remember?) from this file.
The weird thing about auto generated stuff however is that you don’t know when they get out dated, and I don’t think you should. Let’s write a test to handle this.
I tend to write this kind of test in a file right next to the services file. In digest the file is named digest.go in the root, and therefore:
file: digest_test.go
Now everytime you run your tests (speaking of which, I really suggest making use of goconvey which runs them automatically for you), you’re going to be notified of a possible change in your interfaces’ api that you had forgot to reflect in your mocks by regenrating them.

4. Test generated assets: other things


I really liked go-bindata and still find it quite useful eventhough its been archived. However a problem similar to that of mocks is waiting to hunt you down if you use generated/bundled static assets through language apis. Forgetting to regenrate your bindings can cause problems that are hard to track down and waste some time that we might as well spend watching cat videos on youtube.
This one came really handy in shopapi where I was using graphql. graph-gophers/graphql-go nicely verifies your code against your schema but as I was using go-bindata to feed my schema to graphql-go, I would repeatedly forget to regenerate the static assets, which basically made the library unaware of my new changes to the graphql schema. The solution, as you might guess, is pretty simple:
Now let’s watch some cat videos.

Thursday, January 17, 2019

Changes to my notes in google docs sent to my mailbox


What

Organizing information. I tend to write stuff down a lot. Things that I learn, books I read, etc. and I found out that paper does not allow me to Ctrl-F (or / or Ctrl-S or Ctrl-R. Sorry I don’t ^W) and is hard to carry around. It’s not easy to replicate paper and even worse: I have observed that when you edit one paper copy, the content on the other copies stays the same.
My solution was to write all of my notes in google docs, and when I started doing so I naturally started to complicate what I needed from my digital pen and paper: I hardly ever find the time to go over my previous notes, so why not make it a little easier to read every piece of note that I take at least once?
My solution? I thought that it would be convenient to receive an email containing my newest notes every night.

How

At first I was planning to wrap this into a web app that everyone could use, but I remembered how I think of app developers/users who require/give unnecessary access to personal data, and I decided not to create another information honeypot.
The first roadblock was that google drive api does not allow you to retrieve google docs revisions. That would leave me to either generate the diffs myself, or program to a browser engine to acquire the revisions through the UI (similar to this unbelievably amazing piece of code). I lack superpowers so I chose to go with the former. (But to save my ego I’m going to pretend that I did that to make the solution more versatile)
To access the drive api you have to use oauth2 and therefore need to go through the somewhat complex and slow process of creating a project in your google developer dashboard and enable the drive api key for it. You’ll obtain a credentials.json that contains:
So keep it safe.
You can then download (includes waiting for google drive go api bindings to download the whole universe to satisfy its dependencies), compile and install digest by doing:
After which given that $GOPATH/bin is in your $PATH, you’ll be able to execute digest --help.
Remember that credentials.json file? put it here:
Your first execution will probably look something like:
where $GOOGLE_DRIVE_FOLDER_NAME is the name of the folder which contains all of your notes and preferably nothing else. (If you are organized (read: obsessive) enough to write everything in Google docs, and more importantly are willing to review what you write in a systematic manner, I assume that you have 10+ levels of nesting in your documents structure. Just use the name of the top level folder).
Also viper allows you to seamlessly accept environment variables for the flags:
Which means $DIG_SMTP_PASS conveniently can be set as an environment variable instead of passing in --smtp-pass=yourpass.
And that’s it. This first execution takes the first snapshot, and does not yet send you an email. Your subsequent usages will probably look like this:
This will send you an email containing the diff with the last snapshot. A snapshot is taken (and put in ~/.digest/data/) whenever digest is successfully run. If no difference is found between two consecutive snapshots, you’ll enjoy some free panic in your email:

And the content of the email gets a little bit more interesting when you’ve actually applied some changes to your docs:
I issue the $ digest command on my pc everyday, just before I call it a night and read the emailed diff on my phone on my way back home. However if you only hang out with the very cool kids you may already be thinking about:
Feed it to the cron engine on your ser… who am I kidding. If you’re into that kind of stuff you already know how to work your magic.
Anyway, if digest fits in your routine, do tell me about it! Specially if it involves crazy automation scenarios (Bonus points if you manage to trigger it with your office door lock).
UPD: Again if the following condition is true about you, you won’t need no mortal’s guidance but if you’re a systemd freak, you may want to use hooks to digest your notes before a shutdown/hibernate. Hint: DefaultDependencies, oneshot.

Internals

Here’s part of the project directory structure:
├── cmd
│   └── digest
│       ├── digest.go
│       └── main.go
├── diff
│   ├── diff.go
├── drive
│   ├── auth.go
│   ├── drive.go
├── smtp
│   ├── smtp.go
I have implemented the following services in their respective directories and have hooked everything up together (a little uglier than usual I must say) in cmd/digest.
I have used go modules for maintaining the dependencies of this project. If you haven’t heard of them I suggest starting at Russ Cox’s talk on the subject.