Guilherme Garnier

Please stop calling the pandemic a "remote work experiment" | Guilherme Garnier

Tue, 13 Apr 2021 00:00:00 +0000

Since the Covid 19 pandemic started, we’ve seen the whole world start “accepting” remote work - I put accepting in quotes because most companies don’t really embrace it, but don’t have an alternative. With that, I’ve also seen some studies analyzing the results of this “experiment” - stuff like “people who started working remotely in 2020 are x% more/less productive”. Please don’t do that.

Experiments should, by definition, be prepared to run in a controlled environment. Bringing to the software development world, it’s like using a containerized environment: an ephemeral, isolated process, running in its own namespaces. This avoids interferences from outside - like dependencies with different software versions and port conflicts.

We’re living a pandemic. This is the complete opposite of an experiment. If there’s one thing we don’t have any more is control. We don’t control where we can go, what we can do, and we have no idea when this is going to end. And many of us are suffering with mental health issues. That means we can’t compare the productivity of remote workers now with office workers before.

Most people worked physically in their companies offices, close to their co-workers, easily separating work and personal life: you leave home in the morning and start your work; you get back home at the end of the day and you’re back to your personal life. And most of these people work for companies that weren’t prepared for remote work - that means much more than simply closing the offices and sending employers home. All the work experience must change - communication, interactions, meetings, feedbacks. I’m not saying it’s better or worse, it’s just different. So if you try to reproduce the office environment remotely, you’ll probably fail.

My conclusion is that trying to compare productivity from office workers before the pandemic with remote workers during it doesn’t make sense, for two main reasons.

The first one is that an experiment should have a preparation step. If a company decides to start a remote work experiment, it should study how remote companies work, and how they should change their dynamics. And we didn’t have time for this when Covid 19 pandemic started. Some of these people may not have a decent work environment at home; maybe, they live with noisy relatives; maybe they have kids, and now have to handle remote teaching for their kids and adapt them too, changing their lives completely. And besides all that, they still have to be productive at work!

The second reason is that, anywhere you work, this pandemic is messing with everyone’s lives. People are worried, anxious, depressed. People lost their relatives and friends, and may be worried with what could happen to them. And nobody could ever be fully productive with this situation. If you’re suffering with any of these, as most of us are, you should look for professional help, and not blame remote work.

All this shows that, to focus on work, people need to handle their mental health, and it’s hard. Some companies have realized this, and are taking good care of their employees - including Jusbrasil, where I work. But others are more worried about losing productivity when working remotely. Please don’t be one of the latter.

How to check open ports by process | Guilherme Garnier

Tue, 29 Oct 2019 00:00:00 +0000

When debugging a server problem, you may need to know which ports a specific process opened - or which process is listening on a specific port. There are many ways to do this, using different tools. It may be useful to know more than one, because you may not have every tool available (specially in containerized environments).

Checking if a port is open

The first step is checking if a port is open. The easiest way to do this is using netcat. In this example, there’s an HTTP server running on port 8000:

$ nc -zv localhost 8000
Connection to localhost 8000 port [tcp/*] succeeded!

$ nc -zv localhost 8001
nc: connect to localhost port 8001 (tcp) failed: Connection refused

You can also add a -u flag to check UDP ports.

Another tool you can use to check for open ports is telnet:

$ telnet localhost 8000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

$ telnet localhost 8001
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Telnet can’t be used to test UDP ports.

Testing HTTP ports

In the specific case of an HTTP server, you can use curl to make requests - -v flags makes the output verbose:

$ curl -v localhost:8000/check
*   Trying 127.0.0.1:8000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /check HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 2
< ETag: W/"2-4KoCHiHd29bYzs7HHpz1ZA"
< Date: Fri, 13 Sep 2019 21:11:22 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
OK

When the port isn’t open, you get a Connection refused message:

$ curl -v localhost:8001/check
*   Trying 127.0.0.1:8001...
* TCP_NODELAY set
* connect to 127.0.0.1 port 8001 failed: Connection refused
* Failed to connect to localhost port 8001: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8001: Connection refused

You can also make this same request with telnet, if curl is unavailable:

$ telnet localhost 8000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /check

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 2
ETag: W/"2-4KoCHiHd29bYzs7HHpz1ZA"
Date: Fri, 13 Sep 2019 21:16:53 GMT
Connection: close

OKConnection closed by foreign host.

Checking open ports by process

Sometimes you need to check which ports a specific process is listening to. First you need to find out the process id (pid) using ps:

$ ps aux | grep python
ggarnier  8762  0.6  0.0  94900 14928 pts/2    S+   17:31   0:00 python server.py
ggarnier  8789  0.0  0.0  21536  1092 pts/11   S+   17:31   0:00 grep python

The process id is the second column - in this example it’s 8762. Now you can use lsof:

$ lsof -Pan -p 8762 -i
COMMAND  PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
python  8762 ggarnier    3u  IPv4 13254927      0t0  TCP *:8000 (LISTEN)

In the example above, process 8762 is listening on port TCP 8000.

An alternative command is ss:

$ ss -tulpn | grep "pid=8762"
tcp    LISTEN   0        5                                      0.0.0.0:8000                  0.0.0.0:*            users:(("python"id=8762,fd=3))

You may need to run lsof and ss with sudo if the process is owned by another user.

Checking which process is listening on a port

Finally, if a port is open and you don’t know with process did it, use lsof:

$ lsof -i :8000
COMMAND  PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
python  8762 ggarnier    3u  IPv4 13254927      0t0  TCP *:8000 (LISTEN)

netstat can also be used. It doesn’t support filtering by port or process, but you can use grep to filter its output:

$ netstat -ntlp | grep 8000
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      8762/python

In both commands, use sudo if you aren’t the process owner.

Build a print server with a Raspberry Pi | Guilherme Garnier

Thu, 07 Feb 2019 00:00:00 +0000

A few months ago, my wife needed to print a couple of documents. I had an old HP printer (Officejet 4355) sitting around, but I couldn’t use it in her laptop - it’s a Chromebook, which doesn’t support printer drivers. It can only print over the network, and my printer doesn’t have network connection, just USB. The only way I could make that work was installing the printer in another computer and making it share the printer over the network. But I’d have to keep the other computer on and connected to the printer every time she wanted to print something, which isn’t convenient.

So I decided to remove the dust from an old Raspberry Pi (the original model B) I hadn’t used for a while, and use it as a print server! Now I can print from any device, including her Chromebook and my Android phone.

Installation

First install Raspbian (without graphical interface, as you’ll use it only as a server), and set a static IP for it. Then connect the printer to one of its USB ports and install the driver - for my case, I found HP’s Linux Imaging and Printing software (HPLIP). It’s an official HP driver for Linux that supports many old printers, including mine. To install it:

sudo apt-get install hplip

The next step is installing CUPS and adding your user (default user is pi) to lpadmin group:

sudo apt-get install cups
sudo usermod -a -G lpadmin pi

To add a printer to CUPS, open its web interface in http://<pi IP address>:631/admin and click Add Printer. Your printer should be discovered automatically. After selecting it, you can add a meaningful name to it. Also, don’t forget to check Share This Printer before continuing.

With the print server fully configured, you need to setup the printer in each device. In a laptop running Ubuntu, my “new” printer could be automatically discovered. If you have any problem with it, check Debugging Printing Problems. In Chromebook, I installed IPP/CUPS printing extension. And in Android, I’m using CUPS printing app. They all work great!

The anatomy of a code review | Guilherme Garnier

Tue, 08 May 2018 00:00:00 +0000

Code review is an amazing tool to improve code quality. It brings many benefits, to the reviewer, to the reviewee and to the team.

How we do code reviews

To add code reviews to your team workflow, first you need to use a version control tool. If you don’t, start with that.

Basically, the idea is never committing directly to the master branch - or whatever other branch you choose as the main one, but usually it’s the master branch. To start any changes to the source code, be it a new feature, refactorings or bugfixes, you need to create a new branch. You can also create conventions for the branch names, but this is not required.

After you finish working on your own branch, you can trigger the code review process. If you use a repository manager like GitHub or GitLab, they already provide tools to help on this process: GitHub calls them pull requests, and GitLab, merge requests - to make it simpler, I’ll call them change requests from now on. You can add some comments explaining the purpose of the proposed changes, and maybe links to other related change requests or issues. These are very important to make the objective of that change request clear for everyone that might review your code. When in doubt, be as specific and clear as you can. Here’s an example from tsuru, the project I currently work in: a pull request with a detailed description.

As soon as a change request is created, it’s available for any other member of the team to start reviewing it. You can also assign it to a specific person, if you think his/her opinion is important - maybe when you change a specific and obscure part of the project that not everyone dominates.

As a reviewer, you job is to check many aspects of the code. Here’s a non exaustive reference list:

accuracy and completeness: does the code do what it’s supposed to do?
bug free: doesn’t it introduce new bugs (like not handling corner cases)?
expressiveness: is it clear on its intentions? Can a reader clearly understand the coder’s objectives?
scalability: could it handle the expected load?
conventions and standards: does it follow the agreed conventions for code style, file and directory structure, etc?
tests: do the tests cover all the use cases, including important corner cases? Is there a specific use case that deserves more tests?

After checking the code, the reviewer can basically accept or reject it - GitHub also allows you to add review comments without explicitly approving or rejecting. When you approve, that means you think those changes could be promptly merged to the main branch. But when you reject, that could mean you doesn’t agree with that change at all (e.g. you don’t think this software should have the proposed feature), or that you’re requesting some changes to the code. That’s what review comments are meant for.

Usually a review comment can be added to a specific line or code block. That could be a typo, a function you think should be named differently, a bug you spotted or a missing test or documentation. The comments should make this clear to the reviewee, who can reply to the comments or make requested changes. Then, this process repeats until the change request gets approved.

When the reviewer accepts the proposed changes, the reviewee has two options: merge/rebase the changes to the main branch and finish the code review process, or ask for another developer review. This is the case when the change is very complex and you’re insecure you may be missing something. When this happens, the review process doesn’t finish until all reviewers accept the changes.

To illustrate, here’s another example of code review from tsuru: in this pull request, I received a couple of comments with request changes. For most of them I changed the code according the suggestions, and one of them generated a small discussion. What’s best is that the discussion is documented in the pull request, for anyone to read and participate.

Benefits of the code review

A code review process brings a lot of benefits to the reviewer, to the reviewee and to the team as a whole:

For the reviewer

learn about the changes: if someone commits directly on the main branch, only the author would know the details about it. When you review, you also get to know what those changes do, how and why they were written. And if you don’t understand, you have the opportunity to ask for more information
learn about the project: if you are new in the team, code review is a great tool to obtain knowledge about the project. Even if you still don’t understand the project, go ahead and review everyone’s code, just to start understanding it a little better
learn about the technologies involved (languages, frameworks, libraries): just like when you’re new to the team, if you are getting started with the tech stack used in the project, you’ll benefit even more from reviewing people’s code. You’ll learn about the language’s features, new libraries to solve common problems and features of the frameworks the project uses
develop a critical view of someone else’s code: if you are an inexperienced developer, you probably have a hard time trying to spot problems in someone else’s code. After code reviewing a couple of times (and checking other developer’s reviews), you’ll learn how to be more critic. And what’s more important, you’ll learn that criticizing someone else’s code is not offensive, it’s actually a good thing, an opportunity for them to learn. And you’ll learn with the other’s mistakes. That’s why junior devs should review seniors’ commits

For the reviewee

learn about the project: just like the reviewer, a great way to start in a new project is asking for feedback from the more experienced in the team. Here’s another example
learn about the technologies involved (languages, frameworks, libraries): the same comments from the reviewer part are worth here. Here’s one more example
learn other ways to solve problems: when you face a bug, you may already have the complete solution in your head; that solution could really work after you implement it. But that doesn’t mean this is the only way to solve the problem. Probably there are other solutions out there, and they may be simpler, clearer, safer or perform better. Other developers could show these other solutions in their reviews
learn to accept critics: sometimes we get so attached to the code we wrote that we could get offended when someone criticizes it. Code reviews help us learn to get over this, because we are explicitly asking for people to give feedback on our code, and they’ll answer. Over time you’ll learn these feedbacks are great ways of learning

For the team

shared code ownership: now two or more people are responsible for the code one of them wrote - by the way, that could also be reached with pair programming. Shared code ownership is great after a bug is released into production: the developer who wrote the code is no longer the only responsible for that. This also helps reaching a blameless culture
helps keeping standards: if the team agreed on specific conventions, like code style rules, tests for every line of code or document all the APIs, code review is a great way to make people monitor each other on these

Common problems and concerns

Won’t this slow me down?

When we suggest that our team starts working with code reviews, people usually get apprehensive about this process making the team slower. If they’re used to finish their work, push the code and deploy to production, they may have this concern.

In fact, you won’t have that fast dev-to-production cycle. But that is a small downside against the many benefits presented above. It’s a trade-off, like every decision you make in your project.

One common concern is having to wait for a long time for another dev to start the code review process. If people still don’t see value in this process, they indeed may not prioritize this. But as soon as you get to be on the other side - waiting for other people’s reviews -, you’ll start giving it more attention. It’s an organic process that usually regulates itself.

How to handle large change requests?

Another problem is when the change request is very large. This will require more time from the reviewers, and it may be harder to analyze: you may lose focus during the review.

Just like on the previous item, this should be self regulated. If you start making very large change requests, you’ll learn they aren’t much productive when someone else asks for you to review a large change request.

There isn’t an ideal size of change request. But you should make them as small as possible. If you’re working on a complex feature, you may try splitting it in a couple of small change requests, independent from each other. But if you can’t make them independent, an alternative is starting with a branch forked from the main one - you may call it feature-something, then creating other branches from it. As you finish each part, make the change requests to merge to you feature branch, not the main branch. And only when you finish every “sub-feature” change request, only then you merge your feature branch to the main one.

When not to use code reviews?

Finally, another common question is: are there exceptions to the “never commit to the main branch” rule? If the change is very simple and small, won’t the code review process be just a formality?

In fact, there are a couple of situations where you may bypass the code review process. One example is updating a dependency version. But I still think it’s worth opening the change request, to make other team members aware of the changes. But this kind of decision is up to the team, and should be accorded among them.

To finish, I suggest reading this great post about how to conduct effective code reviews.

Increasing productivity in tmux with a single prefix key | Guilherme Garnier

Tue, 05 Dec 2017 00:00:00 +0000

Tmux is a fantastic tool for improving productivity when working with a terminal. One of the first things people configure when start using tmux is changing the prefix key. The default value is control+b, which is not very confortable to press with a single hand. And as you’ll end up pressing it a lot, for every tmux command, the most common used configuration is changing it to control+a.

This is much better, but you still need to press two keys simultaneously before typing any tmux command. After using this configuration for some time, I decided to change it to a single key, to make it even easier.

I though about changing the prefix to caps lock. Besides being rarely used, it’s very well positioned. However, you can’t set caps lock as prefix in tmux. An alternative solution is mapping the caps lock key to something else. In OSX, you can set it to another modifier key, like control, shift or esc: go to System Preferences => Keyboard => Modifier keys. First I tried mapping it to esc, and setting esc as tmux prefix. It works, but this setup brought another problem: as a vim user, I use the esc key a lot (to alternate between vim modes), so now I had to type esc/caps lock twice to send the esc key to vim. It was ok, but not ideal.

Then I tried another solution: I installed Karabiner-Elements, a Mac app which allows you to completely customize your keyboard. So I mapped the caps lock key to Home (which doesn’t exist in Mac keyboard), and changed tmux prefix key to Home:

set -g prefix Home
unbind C-b
bind-key Home send-prefix

Now I have a great configuration: I use a single key (caps lock) as prefix, and without losing any key functionality.

UPDATE: to do this same configuration in Linux, you just need to open the /usr/share/X11/xkb/symbols/pc file and change the line that starts with key <CAPS> to this:

key <CAPS> {        [ Home          ]       };

Cheap and reliable archiving with Amazon Glacier | Guilherme Garnier

Wed, 12 Jul 2017 00:00:00 +0000

Incidents like WannaCry ransomware expose the importance of doing backups, which is usually forgotten by many people.

When someone talks about backing up our personal files, we usually think about services like Dropbox and Google Drive. But they have a reasonable cost if you have more than a couple of GB of data. There are a lot of other solutions available. Most of them are cheaper, but not always reliable - imagine if you backup all your personal data to a small and unknown service, and a few months later, the company breaks. Or a security flaw exposes all your personal data! Of course this could also happen with Dropbox and Google Drive, but it’s much less likely, being two large and serious companies.

One alternative to them is Amazon Glacier. It’s a not so popular Amazon service for data archiving. You should notice it works differently from the usual backup solutions. When you sign up to Dropbox, for instance, you can install an app to your computer or mobile phone, or use the web interface to instantly access your files and upload new ones. Glacier is much more low level. It doesn’t have a web interface, app or even command line tool! There’s only an API, which you use to check your files, download or upload.

And there’s more: the download and upload rates are very slow. And to download a file, you first have to ask for a file retrieval job; the download will be available in a couple of hours!!!

This seems like a terrible service, so why use it? Because it’s very, very cheap! You only pay US$ 0.004 per GB per month for storage, besides additional costs for requests. And even being slow and hard to use, it’s a service offered by Amazon, which gives you confidence it won’t suddenly disappear.

Having said that, Glacier isn’t a service to keep data you may need immediately. But it’s ideal for something you probably won’t need to access anytime soon. Think about your family pictures: when you want to access them, you probably doesn’t need them right away; you’re fine waiting a couple of hours for that.

Glacier is also a great option for “backups of backups”. If you want to be neurotic about backups (and you should!), you can archive a copy of your backups there.

Usage

The easiest way to use Glacier is with a third party client. I like amazon-glacier-cmd-interface. After setting up the basic configuration, you can create a vault and upload you files:

$ glacier-cmd mkvault my-disaster-backup
$ glacier-cmd upload my-disaster-backup my-file1 my-file2 ...

To list archives in a vault:

$ glacier-cmd inventory <vaultname>

The inventory retrieval job takes a couple of hours to be processed. You can check its status with:

$ glacier-cmd listjobs <vaultname>
+------------------------------------------------------+---------------+--------------+--------------------+--------------------------+------------+
|                      VaultARN                        |    Job ID     | Archive ID   |       Action       |        Initiated         |   Status   |
+------------------------------------------------------+---------------+--------------+--------------------+--------------------------+------------+
| arn:aws:glacier:us-west-2:483413266890:vaults/backup | QYqdvM4k8q... |    None      | InventoryRetrieval | 2017-07-24T15:47:48.310Z | InProgress |
+------------------------------------------------------+---------------+--------------+--------------------+--------------------------+------------+

When the job status change to Succeeded, run the inventory command again to check the archive list.

To download an archive, first you need to check its id in the inventory:

$ glacier-cmd inventory <vaultname>
Inventory of vault: arn:aws:glacier:us-west-2:483413266890:vaults/backup
Inventory Date: 2017-07-05T11:22:15Z

Content:
+---------------+---------------------+----------------------+------------------+------------+
|  Archive ID   | Archive Description |       Uploaded       | SHA256 tree hash |    Size    |
+---------------+---------------------+----------------------+------------------+------------+
| uFg2FE_guu... | file1.tar.gz        | 2017-03-31T14:29:17Z | b41922e1a2...    | 1342622251 |
| 43Wjk63Dcu... | file2.tar.gz        | 2017-03-31T17:18:28Z | 2346170d22...    | 2347810677 |
+---------------+---------------------+----------------------+------------------+------------+
This vault contains 2 items, total size 2.5 GB.

Then, create an archive retrieval job using the archive id:

$ glacier-cmd getarchive <vaultname> <archive id>
+-----------+---------------+
|   Header  |    Value      |
+-----------+---------------+
|   JobId   | Xa17IAadQG... |
| RequestId | cPcomv_vTf... |
+-----------+---------------+

When the download is available (you can check its status with glacier-cmd listjobs <vaultname>), download it with:

$ glacier-cmd download <vaultname> <archive id> --outfile <filename>

SporTV Play front-end architecture for the Olympic Games | Guilherme Garnier

Tue, 18 Apr 2017 00:00:00 +0000

Globosat Play is a video product for pay TV subscribers, where you can catch up programs you missed on TV. It’s like an umbrella for a couple of different channels. One of the most popular of them is SporTV, one of the largest sports channels in Brazil.

Last year we had the Olympic Games, a major sports event here in Rio. As SporTV channel was going to have a large coverage for the event, we decided to rethink the user experience for SporTV Play (SporTV channel offers inside Globosat Play). In this case, we were going to focus in improving the live TV experience, which is responsible for most of the audience.

Besides rethinking the user experience, we decided to also rethink our front-end architecture, and address a couple of the issues we had.

I’ve already written about Globosat Play architecture before. Its front-end is basically a couple of Rails apps using regular erb templates. These apps share components through a component library called globotv-ui. It’s about 4 years old, way before newer component technologies arised and got popular.

As descripted in the previous post, this component library solution allowed us to avoid rewriting the same components again and again for each app. We were able to share our components, but the main problem is that this was an in-house solution. We defined our components structure to attend our needs, so it was really hard to share them outside our product.

Also, we had a few problems with our front-end architecture. When we updated a component that was used on pages served by different apps, we needed to “synchronize” the deploys. If we deployed one app and took too long before deploying another, in the mean time, both would have different versions of that component. That could generate an inconsistency for our product, and it was one of the issues we wanted to address with the new solution.

In late 2015, we started a couple of technical discussions and some proofs of concept, and decided to adopt React in our front-end. A couple of arguments led the way to our decision:

Standardize components structure

Even after creating a dozen components in globotv-ui, we still could find some differences among them. That’s because the structure is loose and not well documented. Usually we start a new component looking at another one, and replicate that structure. But a lot of different developers worked on that library, each one with his own preferences. So we really didn’t have a pattern for components. They were well organized and tested, but in the end, they were just a couple of JS and CSS files, sometimes with a template for generating the HTML (in Handlebars for client-side rendering, or ERB templates with Rails helpers for server-side).

The clear and well known React component structure helps keeping a pattern among components. The component lifecycle lets us manage how our components should behave. It allows us to not only open source a couple of generic components, but also search for ready components instead of recreating everything we needed.

As an example of this, we wanted to keep our header sticky after the user scrolls down the page. Instead of implementing this behavior, we used react-headroom. Problem solved!

Declarative programming model

Another benefit that React brings is its declarative programming model, instead of the traditional imperative model. Here is a simple example: a text area with a “Tweet” button, which should be disabled while the text area field is empty. Here is the imperative implementation using jQuery:

// Initially disable the button
$("button").prop("disabled", true)

// When the value of the text area changes...
$("textarea").on("input", function() {
  // If there's at least one character...
  if ($(this).val().length > 0) {
    // Enable the button.
    $("button").prop("disabled", false)
  } else {
    // Else, disable the button.
    $("button").prop("disabled", true)
  }
})

Now the React version:

class TweetBox extends Component {
  state = {
    text: ""
  }

  handleChange(event) {
    this.setState({ text: event.target.value })
  }

  render() {
    return (
      <div>
        <textarea onChange={this.handleChange}></textarea>
        <button disabled={this.state.text.length === 0}>Tweet</button>
      </div>
    )
  }
}

Globo Play release

The third strong argument in favor of React was the release of Globo Play, in late 2015. It’s another video product developed here at Globo.com. It’s very similar to Globosat Play, and it already used React. So when we started developing the new interface for SporTV Play at the Olympic Games, the team that developed Globo Play already had a great experience with it to help our adoption.

The new architecture

As descripted in a previous post, our architecture was already microservices-based:

Globosat Play original architecture

In the front-end, we had a couple of different apps to serve different pages in our product, with an nginx server in front of them. The nginx server proxies the requests to each app, according to the request path. Our idea was adding a new rule to it, to forward all requests to SporTV Play home and live signals pages to a new app. This new project was going to use React, and ideally share components with Globo Play.

The first step was thinking about the API for this new front-end app. We already had a couple of APIs serving our current apps, so we didn’t need to create a new API. We could just use what we already had, but we decided to follow the Back-end for Front-end pattern.

The APIs we already had served well many of our apps, and some of them are legacy. They have a lot of services, and most of them are fine-grained. To serve the new front-end app, we would probably need to make many requests to group all the data we needed.

Also, we thought it was a good idea to separate the new app from the old ones. Doing this, we would have less chance of coupling old and new apps. Suppose both of them consumed the same services; that would create a coupling between them. If the new app required a change in this service contract, we wouldn’t be able to do that without the risk of breaking the old app.

Besides that, we know the mobile consumption is raising, and sometimes we suffer with terrible 3/4g connections. With a new API specifically designed to attend the needs of the new app, we could deliver the smallest possible payload (just the data we were going to use). Also, we could create more specific services, reducing the number of requests to a minimum.

We followed two principles from the Back-end for Front-end (BFF) pattern: the front-end consumes services from its BFF and nothing more; and the front-end is the only BFF client. They are tightly coupled together, but this is not a problem, because they should both be maintained by the same team. It’s like we are splitting our app in two. The BFF is responsible for orchestrating requests from internal, fine-grained services, apply some business rules and deliver data ready to be consumed by the front-end, which just consumes these coase-grained services and takes care of the presentation layer.

BFF architecture

One downside of the BFF is code replication. Some of the services we created in the BFF were new and very specific to our new SporTV Play front-end app (like the current schedule and the list of live signals). Others were already available in our internal APIs. But to follow the rule were the front-end can’t access any service outside its BFF, we needed to add a new route to the BFF, and basically make a proxy pass to our internal APIs. As any decision in software engineering, it’s a tradeoff. There is no perfect solution for everything, and the benefits overcome this issue.

For more details and discussions about the BFF pattern, check out Sam Newman’s and Phil Calçado’s articles.

This is the first post of a series. In the next ones, I intend to write a little more about React components, state management, CSS architecture and components sharing.

Handling request cache in Ruby | Guilherme Garnier

Thu, 09 Feb 2017 00:00:00 +0000

Handling HTTP cache is one of the most important aspects when you need to scale a web application. If well used, it can be your best friend; but when badly used, it may be you worst enemy.

I’m not going to explain the basic aspects of caching here, there are already a lot of great material about it. I’m going to bring a specific problem here.

On a previous post, I wrote about Globosat Play architecture. As explained there, it evolved to a microservices architecture, and as such, we ended up making a lot of HTTP requests to our internal services. So we needed to manage those requests very well.

Suppose you access a page like Combate channel home page. To fill up every information in that page, we need to query data from:

a videos API, to bring up a list of available channels and the latest videos from Combate channel
a highlights API, to check the latest highlights selected by an editor
an events API, to check a list of previous and next UFC events

That means a user request could be represented by something like this:

Requests without cache

Now imagine one of these services is unavailable. Or very slow. Or giving un unexpected answer. If we didn’t consider these scenarios, we would end up with a brittle application, susceptible to a lot of issues.

Michael Nygard, in Release It!, says we must develop cynical systems:

Enterprise software must be cynical. Cynical software expects bad things to happen and is never surprised when they do. Cynical software doesn’t even trust itself, so it puts up internal barriers to protect itself from failures. It refuses to get too intimate with other systems, because it could get hurt.

That means we shouldn’t trust anybody. Don’t assume a service is up, available, fast and correct. Even if you know and trust the maintainers of this service, consider that it may have problems (and it will, eventually!). One of the defense mechanisms against this is caching.

At Globosat Play, we decided to implement two levels of cache. We call them performance and stale.

Performance cache

The performance cache is meant to avoid a flood of unnecessary requests to a single resource in a short period of time. Going back to Combate home page example, one of the services our back-end requests is a list of next UFC events. This doesn’t change often; only when a new event is created, or when an event finishes, once a couple of weeks. That means, it’s very wasteful to hit that service for every user accessing Combate home page. Suppose the events API response changes once a week; if that page gets 100,000 hits in that period, that means I would make 100,000 requests for that API, when I could just make one and keep the results in cache, which is much faster.

The solution for this is keeping a performance cache for a specific period of time. Suppose I set my cache for 5 minutes. The decision flow for this would be:

cache available? Respond with cache
cache unavailable? Make the request, write the response in cache, set its TTL (Time-to-leave) for 5 minutes, respond

That means I would hit that API only once every 5 minutes, independenly of how many users are accessing my home page right now. I’m not only avoiding wasteful requests, but also protecting my internal services and giving faster responses - it’s much faster to access the cache than making an HTTP request. The diagram below depicts this scenario:

Requests with cache

The problem in this scenario is, even if I’m sure that my events API only changes once a week, I can’t set my cache TTL for 1 week. Imagine if I do that and the cache expires a few minutes before a new event is registered. That means I won’t see the new event until the next week! You need to carefully evaluate the performance cache times for each service you depend on.

Even if you have a service that can’t be cached for that long, you could have a great benefit from caching the request for at least a few seconds. Imagine an application with 10,000 requests/s. If you set the back-end service request cache TTL for 1 second, you are making a single request for your service, instead of 10,000 requests!

Stale cache

The second cache level is stale. It’s a safety against problems like network instability or service unavailable. Let’s use the latest videos API as an example. Suppose my application back-end tries to access this service and it gets a 500 HTTP status code. If I have a stale cached version of it, I can use it to give a valid response to its client. The stale data may be outdated by a few minutes or hours, but it’s still better than giving no response at all - of course, that depends on the case. For some kinds of services, an outdated response may not be feasible, like giving the wrong balance when your client accesses his bank account. But for most of the cases, stale cache is a great alternative.

Usually we set the performance cache time for a few minutes and the stale cache for a few hours. Our standard setup is 5 minutes and 6 hours, respectivelly.

Implementing cache levels in Ruby

To implement performance and stale cache levels in Ruby applications, we created and open sourced a gem called Content Gateway. With it, it’s much easier to manage cache levels.

After installing it, you need to configure the default request timeout, the performance and stale cache expiration times and the cache backend, besides other optional configurations:

config = OpenStruct.new(
  timeout: 2.seconds,
  cache_expires_in: 5.minutes,
  cache_stale_expires_in: 6.hours,
  cache: ActiveSupport::Cache.lookup_store(:memory_store)
)

gateway = ContentGateway::Gateway.new("My API", config)

With this basic configuration, you can start to make HTTP requests. You can also override the default configurations for each request:

# Params are added via query string
gateway.get("https://www.goodreads.com/search.xml", key: YOUR_KEY, q: "Ender's Game") # => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<GoodreadsResponse>\n  <Request>..."

# Specific configuration params are supported, like "timeout" and "skip_cache"
gateway.get_json("https://api.cdnjs.com/libraries/jquery", timeout: 1.second, skip_cache: true) # => {"name"=>"jquery", "filename"=>"jquery.min.js", "version"=>"3.1.1", ...

It supports POST, PUT and DELETE as well. For all verbs, there are two methods for making the request: one is simply the name of the verb and the other has _json suffix. The former treats the response body as string, and the latter, as a Hash.

gateway.post_json("https://api.dropboxapi.com/2/files/copy", headers: { Authorization: "Bearer ACCESS_TOKEN" }, payload: { from_path: "path1", to_path: "path2" })
gateway.put_json("https://a.wunderlist.com/api/v1/list_positions/id", payload: { values: [4567, 4568, 9876, 234], revision: 123 })
gateway.delete("https://a.wunderlist.com/api/v1/tasks/id")

You can also make a few other customizations. Check out the project page on github for more information and examples.

Software Developers and Digital Artists | Guilherme Garnier

Mon, 12 Dec 2016 00:00:00 +0000

Nerdcast is an amazing podcast (in portuguese) about nerdie stuff in general. One of its latest episodes talked about the digital artist profession. The guests were animators that worked on feature films like Moana and Doctor Strange. They talked a lot about what they do, and I saw a lot of analogies between their work and software development.

One of the podcast guests said that we was working on a specific scene for Doctor Strange for about 2 months, and suddenly his boss called him and told the director decided to cut that scene off from the film. Those 2 months of work turned into garbage. The lesson he learned from that is, you shouldn’t get attached to the project you’re working on. It’s not your project, it’s your company’s project and you happen to be working on it, which is very different. The analogy here is very clear, because that happens a lot in software projects: sometimes your client decides the feature you’ve been working on for a few weeks or months is not that important, or even worse, that the whole project shouldn’t be maintained anymore. It’s very hard not to be impacted by these news, but if you don’t work on that, you will frequently get frustrated. The project is meant to deliver value to your client, not you. Always remember that you are not paid to write code.

If you want a project to be really yours, and have total freedom to decide what to do and how to do it, you need a personal project. But it’s important to notice that, if someday you decide to make that a business, turning that into a startup or selling some kind of product or service, you will start to have clients. When that happens, you need to be ready to let go of your ideas; if nobody wants your product, be ready to pivot (or even discontinue the whole product). A/B tests are also a great way to learn how to let go of your ideas and beliefs: if a hypothesis is proven worse than the default behavior, just delete it.

Another important topic that was mentioned in the podcast was: the guests, as artists, have a hard time deciding when to stop improving their work. They start working on a scene and iterate a couple of times to make it better. As perfectionists, they want to keep polishing their work. But sometimes the scene is already looking so good that any improvement won’t be noticed by the audience, so it just won’t deliver value to the client anymore. The problem is not knowing when to stop. We can make an analogy here with refactoring: sometimes we develop a new feature, and even after it’s implemented and well tested, we decide to refactor. The objective could be making the code clearer for anyone that may touch it later - to implement a new feature of fix a bug - or maybe extracting a part of it to remove duplication from a similar feature you already had. In both cases, the refactoring won’t deliver value in the short term, but will on the mid-term or long term: you will have lower maintenance costs. But we may have the same problem as the animators: it’s very hard to know when to stop polishing the code. At some point, the refactoring won’t deliver value anymore, and we just refactor to please ourselves. As said before, you need to remember the project is not yours, and you are not paid to write code!

Globosat Play architecture overview | Guilherme Garnier

Thu, 01 Sep 2016 00:00:00 +0000

This post is at least one year late. Since I gave a few talks about Globosat Play architecture (slides in portuguese), I intended to write a more detailed post, but always procrastinated about this.

Globo.com is the internet arm of the largest media conglomerate from Brazil, and one of the largest in the world. One of the areas of the company is responsible for our video platform, which includes encoding, distribution and streaming for any website of the group that needs videos.

About 5 years ago, one of our videos team developed a video product called globo.tv (recently discontinued and replaced with a newer product, Globo Play). Its content was focused on Rede Globo (our main broadcast TV network) shows, like news, sports and the famous brazilian telenovelas. Most of globo.tv content were small scenes from these TV shows, open for every user, but it also offered full episodes for paying subscribers.

globo.tv home page

The original architecture was a single monolithic Rails app, with a Unicorn application server, a MongoDB database and a Redis instance for cache, all behind an nginx HTTP server. This architecture served very well for some time.

globo.tv architecture

Then came 2012 with new demands for globo.tv. We needed to start offering live streaming of a couple of events, like UFC, Big Brother Brasil, soccer championships and the Winter Olympic Games. Also, we needed to start offering a collection of videos from Combate channel, focused on MMA sports.

At this point, we realized the old monolithic architecture wouldn’t serve anymore. So we needed to break it in smaller parts. The first step was identifying smaller subdomains inside videos. The first ones we identified and split were live streaming, for these new live events demand, and VoD (video on demand), for Combate channel videos.

Clearly these new subdomains deserved their own projects, but the problem is, they needed to share a few business rules and data from the original globo.tv project, and also between them. That meant we needed to extract a few services from globo.tv to its own project: globotv-api was born!

globo.tv architecture, 2.0

At this point, we had:

globotv-api, which offered a few basic video services, like the most recent videos from a specific program and the most watched programs
globotv, the remain of the original project, which started to consume globotv-api services. It was responsible for serving the original pages, like home, program page, video page and search
globotv-events, a new project responsible for live streaming of different kinds of events. It also consumed globotv-api and offered its own API with specific services, like a list of live events happening right now
globotv-vod, another new project, which served the collection of videos for Combate channel. It also consumed globotv-api and offered its own API with specific services, like a list of competitions and videos from a specific fighter

The break-up of the monolith was a very important move. If we didn’t split it at that time, we would end up with a larger and larger project, which would soon become a monster - harder to understand, harder to maintain, harder to evolve. It allowed us to share a small part of our domain, which was common among these new requirements and the original project.

To split the front-end among different projects, we used nginx as a router. We already used it in front of our application server, but with multiple projects, we created an upstream for each one, and configured a location for each URL pattern, like this:

upstream globotv {
  server globotv.internal.globo.com;
}

upstream globotv-events {
  server globotv-events.internal.globo.com;
}

upstream globotv-vod {
  server globotv-vod.internal.globo.com;
}

server {
  listen 80;
  server_name globotv.globo.com;

  location ~ (.+)/ao-vivo/ {
    proxy_set_header Host $http_host;
    proxy_pass http://globotv-events;
    break;
  }

  location ~ ^/combate/ {
    proxy_set_header Host $http_host;
    proxy_pass http://globotv-vod;
    break;
  }

  location ~ /  {
    proxy_set_header Host $http_host;
    proxy_pass http://globotv;
    break;
  }
}

The internal domains aren’t publicly exposed, the only way to access the projects is through nginx. With a configuration similar to this, nginx routes incoming requests for globotv.globo.com domain to different projects, according to the URL pattern: URLs that contain “/ao-vivo/” pattern (“live” in portuguese) are routed to globotv-events project; URLs starting with “/combate/” are forwarded to globotv-vod project. The last location matches every other URL to the original globotv project. globotv-api project doesn’t appear in this configuration, as it isn’t publicly accessible (it’s only accessed from the other projects). This configuration allowed us to serve different pages from different projects transparently for the user.

Despite many benefits, this architectural change brought new challenges. The first one was keeping a consistent visual identity among pages served from different projects. For example, the video thumb component used in the home page should be just like the video thumb from the search page; the product header should be the same in every page. The split in multiple projects was an architectural decision; the user doesn’t need to know this, because globo.tv was still a single product.

The solution for this problem was creating a components library, called globotv-ui. With this solution, we were able to share visual components, comprised of HTML, JS and CSS. They were standardized and documented, which made it very easy to create new components and share them among these projects - as all of them were Ruby on Rails projects, we delivered globotv-ui library as a Rubygem.

examples of components from globotv-ui library

Fast forward a few years, and a new video product emerged: Globosat Play. It was very similiar to globo.tv on a few aspects - the idea was offering VoD and live streaming from Globosat channels (TV channels available only for paying subscribers), but with a few differences: we also needed to offer movies, and now the focus was on subscribers, and not on free users anymore.

Globosat Play home page

The main challenge at that point was how to share components and services between these two products, but without one limiting the other’s evolution and requirements. We needed to re-evaluate our architecture and business domain to solve this issue. We realized that, as both products had many similarities and some differences, we needed to create a common, shared layer of services, but also keep some services specific. The new architecture has become something like this:

Globosat Play architecture

This diagram is simplified from the previous ones - the new projects are also Rails apps with their own MongoDB and Redis instances. The green boxes are projects that serve web pages, and the blue ones are APIs. We split our projects in 3 parts: the top one in the image is specific to globo.tv product; the bottom one is specific to Globosat Play; and the middle one represent shared services. You can notice most of the projects created before (globotv-api, globotv-events and globotv-vod) started being shared, as Globosat Play also had those same requirements.

Besides that, we created a few other projects. Some of them, like movies, attended a specific subdomain that didn’t exist before; others, like globotv-search, were extracted from the original globotv project - these features already existed, but now we needed to share them between globo.tv and Globosat Play. Also, globotv-api kept being our main source for basic video services.

This evolution also required a few new configurations on our nginx server:

upstream movies {
  server movies.internal.globo.com;
}

upstream globotv-search {
  server globotv-search.internal.globo.com;
}

upstream globosat-play {
  server globosatplay.internal.globo.com;
}

upstream globotv-events {
  server globotv-events.internal.globo.com;
}

upstream globotv-vod {
  server globotv-vod.internal.globo.com;
}

server {
  listen 80;
  server_name globosatplay.globo.com;

  location ~ (.+)/ao-vivo/ {
    proxy_set_header Host $http_host;
    proxy_pass http://globotv-events;
    break;
  }

  location ~ ^/telecine/ {
    proxy_set_header Host $http_host;
    proxy_pass http://movies;
    break;
  }

  location ~ ^/busca/ {
    proxy_set_header Host $http_host;
    proxy_pass http://globotv-search;
    break;
  }

  location ~ /  {
    proxy_set_header Host $http_host;
    proxy_pass http://globosat-play;
    break;
  }
}

In the end, we realized the microservices architecture brought a lot of advantages:

smaller, easier to manage projects: each subdomain is separated in its own project, which helps keeping the code smaller and easier to understand. Also, this allows different teams to work in different projects
faster builds: with many small projects, the time needed to build and run test suites for each one gets smaller, which gives faster feedback cycles. That stimulates developers to run the test suites more frequently
smaller and less risky deploys: each project is responsible for a small part of the product; that means bugs only affect that small subdomain of it. Suppose you introduce a bug in the search project; that would affect only the search page. Your users would still be able to access the home page and watch videos. That gives confidence for the team to deploy to production more frequently, which reduces even more the risk of bugs
flexible infrastructure: every service is a REST API over HTTP, using JSON format. That means you could write each one in a different language with a different database technology, if you wanted and needed. You could select the best tools for the job
easier incremental changes: suppose you want to migrate your application server from Unicorn to Puma. With a monolithic application, you would need to flip the switch all at once. With many small services, you could choose one to try Puma - maybe the less critical one. If the migration is successful, you could continue with this process one project at a time

But we also had a few disadvantages:

more complex architecture: when a new developer starts in your team, it’s much harder to explain to him how the architecture works and what each project is responsible for
harder local environment setup: another problem for new developers is setting up the local environment. With many projects, each one with its own requirements, that’s much harder
harder to update dependencies, like newer gems: when you need to update a dependency, like a gem that fixes a critical security flaw, you need to do that once for each project. With a monolith, you would just need to do that a single time
harder to test: when each service depends on many others, the setup for the test environment is much harder. You would need to set a complex environment for integration tests, or create fake APIs to replace real ones, or maybe mock API responses, using something like VCR
heterogeneous environment: the flexibility earned with microservices might result in projects in different languages, with different databases and other dependencies. That makes it harder to maintain, because not every developer may understand the whole set of technologies
more failure points, harder to debug: when a service fails, the problem may be a bug or database failure, for example, but may also be the result of a failure in another service it depends on. The debugging process gets harder and slower

In the end, the migration from monolith to microservices was very successful to us. But the main point here is realizing that microservices aren’t the magical solution to every problem - a messy monolith split would generate many messy microservices, as illustrated in the image below. You should analyze the characteristics of your project before deciding if microservices are the best option.

Monolithic vs Microservices pic.twitter.com/EFUpsYkBNu
— Fuckowski (@fuckowski) April 6, 2016

Modern-world birthday statistics | Guilherme Garnier

Thu, 18 Feb 2016 00:00:00 +0000

When I was younger, I used to get a lot of phone calls during my birthdays, from my relatives and friends. But now the situation is much different.

A few days ago I got older, and the “happy birthday” wishes came from many different mediums. So I decided to compile some curious (and completely useless) statistics about this:

Sometimes technology makes life more complicated, instead of simplifying it.

Software quality isn't optional | Guilherme Garnier

Thu, 05 Nov 2015 00:00:00 +0000

In the last weekend, I attended to DevDay 2015, in Belo Horizonte, where I presented a talk about “evolution of a distributed architecture” (in portuguese).

The last talk of the event presented Stack Overflow architecture. This talk was very controversial, and I felt I needed to write something to show my point of view on this subject.

A lot is being said in the community about the value of software engineering best practices. For the last decades, people like Uncle Bob, Kent Beck, Martin Fowler and many others have been doing a great effort promoting these practices, through books, posts and talks. These professionals made and still make a great job orienting new developers, so we can have well tested software projects and well designed architectures, focusing on aspects like maintainability, scalability, security and quality.

Despite this, we know that, in many companies, management makes pressure on developers to deliver as fast as possible, forgetting about software quality and maintainability - even knowing they will be responsible for the project maintenance, at least in short term. This habit goes against every best practice described above, and is very harmful to new developers. These ones that start their career in companies with this idea in mind, even if they’ve read about best practices, end up believing that this is utopian, and that in practice it’s impossible to design a sustainable architecture, refactor legacy code or automate tests, because of pressures for fast delivery of new features. That’s why it’s essential to spread the word about best practices, to show these new developers that it’s not only possible to focus on quality, but also essential for the project evolution and maintenance.

In the talk that closed DevDay 2015, Stack Overflow architecture was presented. The displayed numbers are impressive: it’s one of the 50 top sites in the world, with millions of page views per day, and all that supported by only 9 physical servers, each one working at around 5% load, plus 2 database servers. With this structure, the average load time is only 18 ms. How is this possible?

The secret, according to this talk, is their obsession with performance. Every new feature must have the best possible performance. When a library or tool they use isn’t considered fast enough, they rewrite it from scratch. As layered architectures are slow, every database query is manually written, and directly in the controller.

Stack Overflow code has low testability, because you can’t create mocks to replace the database connection, for instance. That’s why the project has very few automated tests (and it was mentioned that some developers don’t even run the tests). As they have a massive number of active and engaged users, any bug that’s introduced after a deploy is quickly found and reported at Meta Stack Overflow. To update the operating system version, they just remove a server from the pool, apply the update and put the server back live. They assume that, if a new bug arises, users will soon find and report it.

The speaker let it very clear that modeling, architecture and tests are good stuff, but they’re not for everyone. I personally disagree.

Stack Overflow is a very particular case. As their audience is made of developers, and they’re very engaged and passionate about the product, bugs in production are considered acceptable, because the main focus is performance. But what’s the point of performance without quality? Would you buy the fastest car in the world even knowing it doesn’t have seat belts and air bags, and that it doesn’t support replacing a flat tire or a defective part? The analogy is very exaggerated - a bug in the site doesn’t involve life risk, but what I mean is, if you focus only and exclusively on performance, you give up other aspects like quality and security. It’s the same line of thought from those managers I mentioned before, who make a lot of pressure for fast delivery, regardless of quality.

Does every fast food need to be like this?

In my opinion, even if your project focuses on performance, quality can’t be abandoned. The other extreme - over-engineering - is also bad; if you have a small application, with only a couple of users, it doesn’t make sense to create a complex architecture, thinking about the possibility of maybe one day it may expand. This would be creating a solution for a problem that doesn’t exist.

I have a real example for this: I register all my expenses in Google Spreadsheet. I wanted to share these data with my wife, but as the spreadsheet is very large, I created a small app that extracts these data and displays a very simple dashboard, only with information that she is interested in. This app has only two users - me and her -, and there isn’t a chance that this grows up. In this case, it doesn’t make sense for me to think about a scalable architecture. But when we talk about a product with tens of millions of users, the situation is very different. Currently, Stack Overflow doesn’t have a competitor to match, but if one rises with a better user experience or new features, they will have a hard time to follow.

My main concern while watching this talk was the impact that these ideas can have on the audience. The majority of them were very young, probably students or professionals starting their career. A talk like this, spreading the word about optional software quality, can be very harmful to them. And it seems like Facebook has a similar issue with code quality.

To wrap up, I want to make it clear that I’m not doing a personal attack against Stack Overflow or the speaker. I admire her courage to take the stage and present such controversial ideas, even though I disagree on them. I’m a Stack Overflow user and will continue being after all.

Lost in translation | Guilherme Garnier

Thu, 05 Nov 2015 00:00:00 +0000

After a few years maintaining this blog, I decided to start writing in english. I feel like I can reach a wider audience with this. Sometimes when I make a comment in a blog post or github issue, or answer a question in Stack Overflow, I want to link to something I wrote in my blog, but as I wrote in portuguese, I can’t do that.

I don’t think this is going to be a problem for most developers - at least I hope so; if you are a software developer and don’t understand english, you should!

Eventually I may write something in portuguese again, if I have a reason for that, so I tagged every old post with portuguese. And every post in english will be tagged with english. I also added direct links in the sidebar, for easy access.

Simulando passagem de tempo em testes de Javascript | Guilherme Garnier

Thu, 10 Sep 2015 00:00:00 +0000

O Jasmine é uma das ferramentas de teste para Javascript mais usadas atualmente. A sintaxe estilo BDD lembra bastante o RSpec, o que facilita a vida de quem já tem experiência com este.

Uma das dificuldades ao realizar testes de Javascript é como simular a passagem do tempo. Existem duas situações básicas onde isso acontece:

Quando o código executa alguma animação, como fade in e slide down, por exemplo
Quando definimos uma função que só será executada após um período de tempo determinado

O primeiro caso pode ser ilustrado com este exemplo básico:

  <button id="button">Show Menu</button>
  <div id="menu" style="display: none">Menu</div>

  <script>
    function example() {
      $("#button").click(function() {
        $("#menu").fadeIn();
      });
    }

    example();
  </script>

Um clique no botão faz com que o menu apareça usando a função jQuery.fadeIn. O teste para este código, a princípio, poderia ser algo assim:

describe("example test", function() {
  beforeEach(function() {
    example();
  });

  it("shows the menu after clicking the button", function() {
    $("#button").click();

    expect($("#menu")).toBeVisible();
  });
});

O problema é que, como a animação do fade in leva um pequeno período de tempo para executar (400 ms por padrão), o menu ainda não está visível no momento em que a expectativa é executada. Uma solução inocente, mas pouco eficiente, para este problema seria executar a expectativa num setTimeout.

Neste caso específico, como a animação é feita usando jQuery, há uma propriedade jQuery.fx.off que permite desabilitar todas as animações. Desta forma, todas as transições são feitas instantaneamente, fazendo com que o teste original funcione:

describe("example test", function() {
  var jQueryFxOff;

  beforeEach(function() {
    jQueryFxOff = $.fx.off;
    $.fx.off = true;
    example();
  });

  afterEach(function() {
    $.fx.off = jQueryFxOff;
  });

  it("shows the menu after clicking the button", function() {
    $("#button").click();

    expect($("#menu")).toBeVisible();
  });
});

Note que o valor original da propriedade é armazenado numa variável e restaurado após o teste, para evitarmos que esta configuração afete outros testes que serão executados em sequencia.

O segundo caso é quando temos algum código que só é executado após um período de tempo - usando setTimeout, por exemplo:

module = {
  someRandomCode: function() {
  },

  waitForIt: function() {
    setTimeout(this.someRandomCode, 5000);
  }
}

module.waitForIt();

A melhor forma de testar este código é “fakeando” a passagem do tempo, para que o teste não precise aguardar. Uma boa ferramenta para isto são os os fake timers do Sinon.JS:

describe("my random test", function() {
  var clock;

  beforeEach(function() {
    clock = sinon.useFakeTimers();
    spyOn(module, "someRandomCode");

    module.waitForIt();
  });

  afterEach(function() {
    clock.restore();
  });

  it("tests my random code", function() {
    clock.tick(5000);
    expect(module.someRandomCode).toHaveBeenCalled();
  });
});

Outra boa opção é usar o Jasmine Clock:

describe("my random test", function() {
  beforeEach(function() {
    jasmine.clock().install();
    spyOn(module, "someRandomCode");

    module.waitForIt();
  });

  afterEach(function() {
    jasmine.clock().uninstall();
  });

  it("tests my random code", function() {
    jasmine.clock().tick(5000);
    expect(module.someRandomCode).toHaveBeenCalled();
  });
});

Ferramentas para avaliação de qualidade de código Ruby | Guilherme Garnier

Thu, 06 Nov 2014 00:00:00 +0000

Ferramentas para avaliar a qualidade do seu código existem aos montes, para qualquer linguagem de programação. Eu já utilizei algumas para Java, mas nunca tinha testado nenhuma para Ruby, apesar de ser a linguagem que mais uso há alguns anos. Por isso, resolvi testar todas as ferramentas que pude encontrar. Separei a avaliação entre serviços e ferramentas.

Serviços

Classifiquei como serviços as ferramentas onde, em vez de instalar e executar localmente, você libera acesso ao seu repositório de código a elas, que coletam métricas a partir do código e geram algum relatório. Os dois serviços que avaliei são gratuitos para projetos open source e pagos para projetos com código fechado.

Code Climate

Este é certamente o serviço mais conhecido para avaliação de qualidade de código Ruby. Seu foco principal é gerar métricas baseadas em complexidade de código, mas ele também é capaz de identificar algumas falhas de segurança e cobertura de código dos testes.

Coveralls

O Coveralls funciona de forma semelhante ao Code Climate, porém tem um foco maior em testes. Ele exibe o histórico de cobertura e a diferença para cada arquivo a cada commit.

Ferramentas

As ferramentas abaixo são open source e distribuídas através de gems. Para utilizá-las, basta instalar a gem e executar um comando, que analisa o código e gera relatórios ou dados brutos para serem analisados.

Brakeman

O Brakeman é uma ferramenta focada em localizar potenciais falhas de segurança no seu código. Ele também exibe alertas especificamente relacionados ao Rails, como falhas de segurança que já foram corrigidas numa versão do Rails mais recente do que a que você usa.

Como ele tem foco em segurança, é muito importante manter esta gem sempre atualizada, para que ele possa detectar falhas descobertas mais recentemente.

RuboCop

O foco do RuboCop é localizar más práticas de programação no seu código, com base no Ruby Style Guide. Algumas das regras são: uso de aspas simples ou duplas para definir strings, tamanho máximo da linha, número de linhas em cada método e uso de espaços na definição de hashes.

Todas as regras do RuboCop podem ser configuradas ou desabilitadas. Basta criar um arquivo .rubocop.yml na raiz do projeto com as configurações desejadas. Ele também possui regras específicas para projetos usando Rails.

Se você usa o editor Atom, também é possível executar o RuboCop automaticamente com o plugin atom-lint. Assim, ao salvar um arquivo, o RuboCop é automaticamente executado, e os alertas são exibidos no próprio editor, ao lado de cada linha.

RubyCritic

O RubyCritic foi criado com o objetivo de se tornar uma alternativa gratuita ao Code Climate. Ele gera um relatório bem semelhante ao deste serviço, reportando os trechos mais complexos do seu código.

Metric Fu

O Metric Fu é um agregador de ferramentas de análise de código. Ele executa diversas ferramentas e gera uma página com links para os resultados de cada uma. É uma das ferramentas mais completas para análise de código Ruby, e uma das mais antigas.

Ruby-lint

O foco do Ruby-lint é localizar erros como variáveis não utilizadas ou não inicializadas, ou número errado de argumentos ao executar um método. O problema é que quando extendemos ou incluimos classes ou módulos definidos fora do projeto (em gems), ele não encontra as dependências e dá erro de constante indefinida. Apesar disso, parece que vem sendo bastante atualizado.

Reek

Esta ferramenta é um detector de bad smells no código, como Long Parameter List e Feature Envy. Possui plugins para rodar alguns editores, como Vim e TextMate.

Roodi

Roodi significa “Ruby Object Oriented Design Inferometer”. Ele executa algumas métricas de complexidade de código, mas é bem básico. A maioria das métricas já são calculadas por outras ferramentas apresentadas aqui.

Flog

Gera um score baseado na complexidade de código. Bem básico.

Rails Best Practices

Ferramenta bem útil, gera métricas de qualidade baseadas no Rails Best Practices. Como diz o nome, é específico para projetos Rails. Também é disponibilizado como um serviço online gratuito, mas somente para projetos públicos no Github. Para projetos privados, é possível instalar uma instância própria, pois o servidor é open source.

Cane

Mais um gerador de métricas de qualidade. As métricas são parecidas com as do RuboCop.

Pelusa

Este projeto é semelhante ao Ruby-lint, mas não consegui executar. Ele só funciona com Rubinius, mas é incompatível com a versão mais recente (apesar de o projeto não informar quais são as versões compatíveis).

Flay

Esta ferramenta procura similaridades na estrutura do seu código (ex: dois métodos que possuem o código muito semelhante).

Saikuro

Gera métricas de complexidade ciclomática.

Laser e Nitpick

Mais duas ferramentas semelhantes ao Ruby-lint. Como estão há anos sem atualizações, nem testei.

Resumindo…

Entre as ferramentas que testei, gostei mais do Brakeman, do RuboCop e do RubyCritic. Acredito que são complementares, e, se usadas em conjunto, ajudam bastante a encontrar falhas de segurança, os pontos mais complexos do seu código e a seguir boas práticas de programação Ruby.

Além destas 3, também gosto do Metric Fu, mas por executar muitas ferramentas, acho que ele gera informação demais. Usando ferramentas e métricas em excesso, geramos tanta informação que acabamos ignorando-as. Por isso, preferi focar nas 3 ferramentas que citei, pois já consigo ter um panorama bem completo do status do meu código com elas.

Usando na prática

Todas as ferramentas que testei são executadas via linha de comando e geram como saída algum tipo de relatório. Apesar de poder executá-las manualmente, na minha opinião, é mais interessante executá-las no servidor de integração contínua (CI). Desta forma, garantimos que essas ferramentas serão executadas com frequencia, e todo o time tem acesso aos relatórios gerados, assim como gráficos de evolução a cada execução. Com isso, podemos analisar se a “saúde” do projeto está melhorando ou piorando, basta acompanhar se o número de warnings de uma determinada ferramenta estão aumentando ou diminuindo.

No caso específico do RuboCop, como ele analisa o uso de boas práticas de programação, acho mais útil executar no editor, pois, ao salvar um arquivo, tenho a resposta imediata, e posso fazer os ajustes no mesmo momento. Mas isso é uma questão de preferência.

Executando no Jenkins

Como uso o Jenkins, descrevi como configurar as ferramentas acima neste servidor. O processo é bem simples, acredito que não seja difícil reproduzí-lo em outros servidores de integração contínua.

Cada ferramenta pode ser executada por um job à parte ou no mesmo job que roda o build e os testes do projeto. Optei pela 1a opção, por 2 motivos:

O tempo de execução de cada ferramenta de análise de código pode ser razoavelmente longo, o que deixaria o job de build muito lento
Caso ocorra algum problema na execução de alguma destas ferramentas, não quero que o job de build do meu projeto apareça quebrado no CI. Se o build e todos os testes foram executados com sucesso, o job que executa o build deve ter sucesso

Além disso, optei por instalar as gems diretamente no CI. Desta forma, além de não precisar configurá-las no Gemfile do projeto, garanto que as gems estarão sempre atualizadas, o que é muito importante, principalmente no caso do Brakeman, pois novas falhas de segurança são encontradas diariamente.

Criando um novo job

Ao criar um novo job no Jenkins, você pode configurá-lo para executar automaticamente após cada build do projeto ou para executar periodicamente. Apesar de a primeira opção garantir que os relatórios de análise de código estarão atualizados a cada build, a execução é um pouco demorada. Além disso, achei que um relatório por dia seria suficiente para acompanhar o status do projeto. Sendo assim, configurei o job de relatórios para executar diariamente, de segunda a sexta-feira. Para isso, na configuração do job, basta selecionar a opção Build periodically, e no campo Schedule, digitar H 0 * * 1-5, por exemplo (o formato é o mesmo usado no crontab). Este valor configura o job para ser executado em qualquer minuto da hora zero, em qualquer dia do mês, todos os meses, de segunda a sexta-feira (dias 1 a 5).

Eu optei por criar um único job para executar todas as ferramentas, pois desta forma, tenho todos os relatórios centralizados num único local. A principal desvantagem é que, desta forma, um erro na execução de uma ferramenta fará o job encerrar com status de erro, e as ferramentas seguintes não serão executadas.

Para configurar cada ferramenta dentro do job, o processo é o mesmo:

Selecionar em Build a opção Execute shell, com os comandos para instalar e executar a gem
Adicionar uma Post-build Action para exibir os resultados

A configuração de cada ferramenta é a seguinte:

Brakeman

Configure a execução da ferramenta digitando os seguintes comandos no campo Execute shell da configuração do job:

mkdir -p tmp
gem install brakeman --no-ri --no-rdoc && brakeman -o tmp/brakeman-output.tabs --no-progress --separate-models --quiet

Para visualizar os resultados, instale o Brakeman Plugin no Jenkins e selecione em Post-build Actions a opção Publish Brakeman warnings. Em Brakeman Output File, digite tmp/brakeman-output.tabs.

RubyCritic

Adicione o seguinte comando no campo Execute shell:

gem install rubycritic --no-ri --no-rdoc && rubycritic app lib

Para visualizar os resultados, é necessário o plugin HTML Publisher. Após instalá-lo, selecione em Post-build Actions a opção Publish HTML reports e digite os seguintes valores:

HTML directory to archive: tmp/rubycritic
Index page[s]: overview.html
Report title: RubyCritic Reports

RuboCop

Digite os comandos abaixo no campo Execute shell:

mkdir -p tmp
gem install rubocop --no-ri --no-rdoc && rubocop --fail-level E --rails --out tmp/rubocop.out app lib spec

O plugin para exibir o resultado desta ferramenta é o Warnings. Após instalá-lo, é necessário configurar um parser para os warnings do RuboCop. Vá até a configuração do Jenkins (Manage Jenkins -> Configure System). Em Compiler Warnings, adicione um novo parser com os seguintes valores:

Name: RuboCop
Link name: RuboCop
Trend report name: RuboCop Warnings
Regular Expression:

^([^:]+):(\d+):\d+: ([^:]): ([^:]+)$

Mapping Script:

import hudson.plugins.warnings.parser.Warning

String fileName = matcher.group(1)
String lineNumber = matcher.group(2)
String category = matcher.group(3)
String message = matcher.group(4)

return new Warning(fileName, Integer.parseInt(lineNumber), "RuboCop Warnings", category, message);

Example Log Message:

attributes/default.rb:21:78: C: Use %r only for regular expressions matching more than 1 '/' character.

Após salvar esta configuração, volte até a configuração do job e selecione em Post-build Actions a opção Scan for compiler warnings. Em File pattern digite tmp/rubocop.out, e no campo Parser, selecione o parser recém-criado, RuboCop.