Carlos Matallín

Building an usable timezone selector (Part I)

2017-08-11T00:00:00+00:00

If you've been around for enough time in the software development field you'll have encountered with the problem of working with date and time data, not to mention timezones.

One of the latest projects I've been working on involved creating a timezone selector. Having a DateTime value stored in UTC format, we needed to be able to represent the data in the corresponding format depending on the timezone selected.

The most straightforward solution to solve this is creating a selector with offset values for each option, which will be added or subtracted to the value stored.

See the Pen timezone-selector-html by Carlos Matallín (@matallo) on CodePen.

The user picks an offset, the date and time are transformed depending on the option selected, and the card is promoted to the next column in the Kanban board. BOOM.

Offset vs timezone

However there is a difference between timezones and offsets. Multiple timezones can correspond to the same offset, and some timezones may change their offset depending on the time of the year.

As the Moment.js guides put it:

A time zone usually has more than one offset from UTC due to daylight saving time. Several time zones may have the same offset at some point during the year. For example, the time zones America/Chicago, America/Denver, and America/Belize all have an offset of -06:00 at varying times. For this reason, it is impossible to infer a time zone from just an offset value.

The Moment.js library provides functionality related to adjusting times based on an offset value, and Moment Timezone library provides the support for adjusting dates based on timezone data.

Making the timezone selector usable

Again, we can just create a selector with the existing timezones, which will be used to calculate the offset for an existing DateTime value, but we need to ask first "How many timezones are there?".

Loading all the data for the Moment Timezone library returns a list of 593 timezones, that can be hard to navigate and pick the prefered option from the selector.

See the Pen timezone-selector-moment-html by Carlos Matallín (@matallo) on CodePen.

Thus, a better question to ask would be "What is the most usable way to show a timezone selector?". We can find several answers in UX Stack Exchange:

Removing the duplicates, showing the offset and some example cities. Example lists of timezones grouped by offset can be found in Wikipedia, or Microsoft documentation.
Offer a text input to search among all the timezone options. This can be achieved with a jQuery plugin such Select2.

These two approaches can be combined.

However there are some drawbacks. Only one timezone value can be assigned to each group of cities, so those cities with the same offset but with different daylight savings time must be splitted into a different group.

Another interesting alternative can be picking the timezone within an interactive map, in the same fashion than macOS, or Ubuntu settings, and even make use of the Geolocation API. Nevertheless, it depends on the constraints in the form of the timezone selector.

Summing up, here are some tips to build an usable timezone selector:

Show timezones, not offsets.
Group timezones with same offset, and show some example cities.
Offer an input field to search timezones.

And the final result:

See the Pen timezone-selector-moment-select2-html by Carlos Matallín (@matallo) on CodePen.

This is the first part of a series of posts where I'll explain how to build an usable timezone selector, stay tuned for the next post.

Blockchain and cryptocurrency

2017-06-04T00:00:00+00:00

Several weeks ago I returned from my vacation in California, and despite I really wanted to publish the notes and pictures I took during those days in the West coast as soon as I came back, I scarcely had time besides work, and training. On top of that a new current obsession has taken over: Blockchain and cryptocurrency.

Blockchain

I first heard of Blockchain several years ago, and mostly referring to Bitcoin. Since then I've been following casually its development, and watching from the sidelines. 1 I wasn't really interested in its use as digital currency, but I was hooked on the underlying technology, and its decentralized design and applications.

If I already lost you, before going any further I'd highly recommend watching this TED talk about How the Blockchain is changing money and business.

Ok, we're back.

If there was a moment I could take a glimpse into the future, and envision what role Blockchains played in it, it was after watching this talk by Mike Hearn: Autonomous agents, self driving cars and Bitcoin. I was already convinced Blockchain was part of that future, but I didn't think we were close yet.

Several years have passed, and Bitcoin still has to face its most notorious challenges: scalability of the platform, whose complexity grows exponentially as Bitcoin does, with bigger and bigger transaction times and fees; and agreement between all parts of the community. There are highly technical problems to solve, too.

Many visible heads have spoken on the current situation of Bitcoin, such Mike Hearn himself, and it doesn't look as bright as once seemed.

Ethereum

Recently, this growing interest has picked up with a new decentralized platform. Ethereum is a distributed computing platform that can execute scripts, or Smart contracts. 2

This new kid on the Blockchain looks more promising because of its capabilities not only as digital currency. While Ethereum popularity is peaking, so is ETH, the currency Token linked to the Ethereum Blockchain, and media outlets like Bloomberg ask if Bitcoin Is at Risk of No Longer Being the Biggest Digital Currency.

The community is thriving, the roadmap has plenty of developments in the pipeline, and every week there are new events and announcements lined up.

Percentage of Total Market Capitalization of Bitcoin and Ethereum

These past weeks have been specially remarkable, and we are in a key moment now:

China's Biggest Exchanges to Add ETH
ETH over 50% of BTC Marketcap
New members joining Enterprise Ethereum Alliance

So I finally decided to take the plunge, and made some small investment.

Investing in cryptocurrency

The first thing to know once you've finally wrapped your head around to throw some money at cryptocurrency is where to get it. There are Exchanges where you can change fiat money (i.e. Euros) for cryptocurrency. My go to option when recommending an Exchange is Coinbase (affiliate link), but there are many others.

The process is very similar in all of them: you create an account, you get verified (in order to prevent money laundering), and you can use your credit card, or make a transfer for cryptocurrency.

Don't take this as Trading Advice, but just use common sense, as in other investments:

Don't invest more than you are willing to lose
Don't try to time the market, you'll only manage to buy high, sell low
Do your research, holding is the best strategy if you believe in the technology
You'd probably want to take out your ETH from the Exchange, and save it in some paper wallet, or buy a hardware wallet
Take into account your capital gain/loss taxes

Once you are in the possesion of some amount of cryptocurrency you will be able to send it between Exchanges, or use tools like ShapeShift to trade any Blockchain asset.

The FOMO is real

Investing in cryptocurrency has been my first approach to any kind of inversion, and FOMO is even worse than social media as you are emotionally, and financially invested.

Now I have adopted a more relaxed attitude, but at the beginning I couldn't help but constantly check the graphs. I also found myself reading all the time to learn the technology, and keeping up with the news in order to make informed decissions.

You'll get familiar with terms like: bull, bear, head & shoulders, margin, long, short, dip, correction, and of course #tothemoon.

If at this point you still have any doubt I'd recommend reading the FAQ in ETHtrader, or the Beginners Guide to Ether Investing.

Tokens, and ICOs

Bitcoin, and Ethereum have inspired the creation of new Tokens, that can be used to incentivize participants in the network. Some people already call Crypto Tokens: A Breakthrough in Open Network Design.

Also, Tokens enable an alternative to traditional financing, where through a token sale, or ICO, anyone can participate in the funding of a product, or a company. This has a huge impact in the technology of the future.

Some examples of these new Tokens, issued on top of the Ethereum network are BAT, from the creator of JavaScript and the co-founder of Mozilla, or Kin, a cryptocurrency from Kik designed to bring people together in a new shared economy.

It's difficult not to think this is some kind of Ponzi scheme or just a fad, but what would you do if you could time-travel to the past and invest in the Internet after seeing its true potential? It's easier in hindsight.

Thanks to Juan Luis who inspired me writing this post.

The closest I've ever been was a friend of mine who made a few grand investing in Bitcoin early. ↩
For a more detailed description here's an overview by Ethereum's creator Vitalik Buterin at Devcon2. ↩

How we work with CSS at CARTO

2016-09-23T00:00:00+00:00

Recently I've switched teams within CARTO and now I'm working full-time in the Builder team, since the CARTO Builder was launched earlier this year. For some years now I've been in charge of the commercial, blog, and docs sites working with a team of talented designers.

While these sites have been redesigned several times during this period until the look they have now after the new brand was introduced, the development and build process has mostly remained, refined in each iteration.

Some back story

In the beginning, the commercial site was part of the user management service which helps runs the CARTO business (aka the backoffice), along with the documentation. The blog lived its own millenial life in Tumblr under a subdomain. The stylesheets for all these sites were scattered between a submodule for this purpose, and the backoffice repository.

This worked for a ~10 people company, but as CARTO grew over 100 people, with dedicated marketing and design teams, it wasn't the case anymore.

One of the first measures taken was splitting the commercial, blog, and docs sites into its own projects. We build them with Jekyll, and then they are deployed to an Amazon S3 bucket via our CI setup, to both staging, and production environments. We even run tests to check the authors and categories are right.

I have a pretty similar setup for this site, which I usually use as a playground for new ideas, and I have been deeply inspired by Yeoman, using Grunt as task runner, and Bower as package manager. 1 Despite that Jekyll manages stylesheets pretty well, with support for SASS, in my opinion there's still need of a better asset management (or space to improve).

How we work with CSS

This just solved the build step, but we still needed some rules, a framework which helped us build the styles and components in the styleguide created by the designers.

There's a recurring discussion that keeps appearing in every team I've been:

How do we work with our CSS?

I have not been able to answer this yet, nor I don't have a silver bullet, but there are some possible solutions. Of course, there's some degree of compromise in every of them.

Styleguide

Most of the complaints CSS (unfairly) receives are related to how easy is to end up with a mess of files, classes, and !important, however there are some guidelines I like to follow to avoid these situations.

Architecture
Use a SMACSS approach for file structure, categorizing CSS rules in Variables, Mixins, Base, Components, Layout, and Utilities.
Preprocessors
Use SCSS as preprocessor. I feel OK using CSS, but the support for mixins, and imports completely increases the productivity working with stylesheets. I'm not totally sold on SASS syntax, though.
Naming
Use SuitCSS naming conventions, or BEM, creating reusable, composable components.
Specificity
Avoid unnecessary nesting. The use of @extend can cause unintended consequences, bloating your files, and adding specificity to your selectors.
Browsers
Use Reset CSS, along with our own default styles. For vendor prefixes use PostCSS.

. main.scss
├── _base.scss
├── _components.scss
├── _layout.scss
├── _mixins.scss
├── _utilities.scss
├── _variables.scss
├── base
│   ├── _defaults.scss
│   └── _reset.scss
├── components
│   ├── _aside.scss
│   ├── _buttons.scss
│   ├── _dialogs.scss
│   ├── _dropdowns.scss
│   ├── _footer.scss
│   ├── _grid.scss
│   ├── _header.scss
│   ├── _icons.scss
│   ├── _navbar.scss
│   ├── _offcanvas.scss
│   ├── _section.scss
│   ├── _separator.scss
│   ├── _subheader.scss
│   └── _titles.scss
├── layout
│   └── _home.scss
├── main.scss
├── mixins
│   └── _clearfix.scss
└── variables
    ├── _colors.scss
    ├── _sizes.scss
    └── _typography.scss

The main.scss file is the entry point, and is the only file which doesn't start with underscore:

/* ==========================================================================
   #MAIN
   ========================================================================== */

/* Non-rendering scss
   ========================================================================== */

@import "variables";
@import "mixins";

/* Base
   ========================================================================== */

@import "base";

/* Components
   ========================================================================== */

@import "components";

/* Layout
   ========================================================================== */

@import "layout";

/* Utilities
   ========================================================================== */

@import "utilities";

By requireing the non-rendering SCSS at the top of the file we make available the variables and mixins to the rest of the files.

Similar files (such components) can be grouped in folders. I prefer to require every file in these folders explicitly.

/* ==========================================================================
   #COMPONENTS
   ========================================================================== */

@import "components/aside";
@import "components/buttons";
@import "components/dialogs";
@import "components/dropdowns";
@import "components/footer";
@import "components/grid";
@import "components/header";
@import "components/icons";
@import "components/navbar";
@import "components/offcanvas";
@import "components/section";
@import "components/separator";
@import "components/subheader";
@import "components/titles";

Minification is applied to the entry files, and sourcemaps are created. Finally optimization tasks are performed like file revision, and gzipping.

There's already some basic linting in the build tasks, and next steps would include extending the check of selectors usage, and implementing Parker for stats.

Development

So, the process of updating the stylesheets would go as follows:

You check if the component you want to use already exists in the current stylesheets, if you need to extend it you create a modifier, if you find two components are very similar you consolidate them into one and create the corresponding children and modifiers, or you just create a new one.

Of course, you'd need to update the components library in every project it is being used.

Distribution

When it comes the time to include the stylesheets in your code, these are the most common scenarios:

Having the stylesheets checked in as part of every project and replicate them where needed. This obviously doesn't scale, and it is very easy to end up having several differenciated user interfaces between projects. Also, you'll find many times having to manually overwrite the styles because they have not been completely updated.
Create a repository and include them as a Git submodule. While this has brought us many headaches in the past, I think it is a good idea if you don't want to fiddle with package managers
Include the CSS via Sprockets. Of course this only applies in some cases like Rails, but I've had great experiences before. When working at Crisalix, we already required our common stylesheets for use with the assets pipeline. GitHub does this with Primer, too, their internal CSS toolkit and guidelines. Edit: Just realized they distribute them via npm now.
Package them with Bower/npm. My go-to option, the good thing is you don't need to publish them, but can install directly from the GitHub URL. They can be bundled in a dist file, and used as a library (as you'll do with Bootstrap), or included separately from the node_modules or bower_components folders, after installing the corresponding package. This is my preferred option to try to keep file size low.

Finally, you can always include them as an external resource after packaging, and deploying it to a CDN. But I would only see this happening for larger projects where there's not direct involvement with the development process.

I'm happy to see UI libraries I built years ago are still being used extensively in production. Some of the examples I currenlty follow:

Do you know any more examples or want to share your experience?

This post has been written using some of my own internal talks resources, and inspired in:

Lately they moved to gulp, but I keep an example generator-webapp using Grunt, which I use as boilerplate for every project. I bet we'll see the same with npm in favour of Bower. ↩

Tokyo

2016-04-18T00:00:00+00:00

Someday, I want to live in that city.

Long story short: I had planned a trip to Japan with my family, but as they couldn't make it for those dates I finally decided to travel solo, inspired by some of my colleagues who had already done so. It's been my first solo trip and I couldn't be happier about the experience. Well, if only I could have extended it.

Day 1Konnichiwa, motherfuckers

Just as I landed, I went to the ATM to withdraw some money as usual, 1 and jumped on the Keisei Bus Airport Express to Tokyo Station. There, I got myself a Suica card, which acts as a prepaid card for transportation and some vending machines. Fares range from ~¥200-800 depending on the distance between stations. ~¥1000 per day worked for me.

Tokyo City Lights

I stayed in Imano Tokyo Hostel in Shinjuku, one of the best districts to get around Tokyo. Both the location and quality have been exceptional. I checked-in, and went out to take a look at the surroundings.

I headed to the Tokyo Metropolitan Government Building, which is free, and you can have some of the best sights of the city. However, it's very difficult to take good pictures due to the flares in the windows. I also visited the Park Hyatt Tokyo, the hotel appearing in Lost in Translation. No doubt that it was the first movie I watched when I came back.

I had for dinner a delicious ramen at Menya Musashi. There's no banner in the facade, only a red curtain indicating the place. I got in, and observed how a local obtained a ticket from the vending machine. I carefully reproduced his steps, and sat in the bar to enjoy my noodles. In the background you can hear some jazz music, and the loud cookers preparing the ramen with some Samurai-like movements.

Something that I noticed on my first days was how well the urban planning seems to work, being one of the most bike friendly cities I've visited.

Day 2Shibuya, Ginza, Akihabara, Imperial Palace, and Shinjuku

Imperial Palace

The next day, I walked from Shinjuku to Shibuya. You can visit Hachiko statue, and the famous Shibuya crossing. There are also big departament stores such Tokyu Hands, where I got a power charger. 2

I took the Metro and went to Designit Tokyo office. 3 I met Agustín Jiménez and we had lunch at a sushi joint where dishes arrived on rails, literally. He recommended me visiting the business district of Ginza, where most banks and luxury brands have their buildings (there's also a Patagonia store). I had a coffee, and improvised a visit to Akihabara.

As much as I loved Tokyo, that district was too much bizarre for me. I gave a quick tour and headed back to the hostel to put on my running wear for a casual run around the Imperial palace, where many runners gather every night.

One of the perks of travelling alone is that you meet people and make new friends. I had dinner with a room mate somewhere in Omoide Yokocho 4 packed with salarymen. The meat was delicious but we went out the place smelling like smoke. BTW! Smoking is allowed at bars. Before calling it a night we had a craft beer in Golden Gai. It was nice, but many pubs charge per entry, or serve an appetizer and charge it even if you don't have a bite.

Day 3Shinjuku Gyoen National Garden, Yoyogi, and Ueno

I was so exhausted from the previous day that I decided to take it easy and stay in bed for some extra time. I woke up at noon and replaced my usual 7-Eleven coffee + doughnut + orange juice breakfast for some Onigiri. 5

It was going to be a relaxed day (or that's what I thought so) visiting parks and admiring the Bloom of Cherry Blossoms. I didn't choose the dates intentionally but I'd strongly recommend to visit Tokyo in these days, not only the parks but the whole city is so beautiful.

Shinjuku Gyoen National Garden, photograph by Carlos Matallín

My first stop was Shinjuku Gyoen National Garden, where I spent most of the day. There you can find a greenhouse, and three distinct styles, French Formal Garden, English Landscape Garden, and Japanese Traditional Garden, which in my opinion was the most impressive featuring cherry trees in all their shades.

Then I went to Yoyogi and had a quick walk. When I was planning my trip I put a pair of running shoes in my backpack, to run in the Oda Field running track, but I'm happy I finally changed my mind, as you are not allowed to run in the Meiji Jingu. I paid respect at the main shrine buildings and catched the Metro to Ueno where I was invited to a wonderful Hanami evening 6 with the Designit Tokyo crew. Just when I thought it couldn't get better we booked a room at a karaoke place.

Welcome to Japanese karaoke.

Drinks, dresses, and songs. I won't forget that night in my life.

Hanami at Ueno Park, photograph by Carlos Matallín

Day 4Hakone

Just a few hours of sleep and thirsty as hell, I got a coffee and ran to Shinjuku Sta. to catch the Romancecar I had previously booked (it hadn't been like that, I would have stayed in bed) to Hakone. The route to the town isn't anything special, but I flipped out when the driver climbed a ladded and disappeared to drive the train, as well as when we arrived and the workers bowed as the train entered the station.

I catched the shuttle bus to Hakone Yuryo, a hot spring resort. Bring your own towel (or buy one at the desk), but leave your swim suit at home. Tatoos aren't allowed either. I spent the morning bathing in hot water butt naked, and had a bowl of hand made noodles at Nisshin Tei Honten.

I really like these kind of days to take a rest from all the frenzy (or recover from the hangover) and plan the next days. Back in Tokyo, I took away a Teriyaki McBurger for dinner on my way to the hostel.

Day 5Tsukiji Market, and JAXA’s Tsukuba Space Center

JAXA's Tsukuba Space Center

I missed the tuna auction, but I didn't want to leave without eating some fresh sushi at Tsukiji Market. Wandering through the sushi stalls was making my mouth water and I queued for several minutes to have lunch at Tsukiji Sushiko. Best sushi I ever had. They have some affordable menus a chef will thoughtfully prepare in front of you.

When I first decided to travel to Japan I didn't know there was a Space Center located in Tsukuba. As you know, I have kind of an obsession with aerospace, so I just couldn't not go. I took the Highway Express Bus from Tokyo Station Yaesu South Exit to Tsukuba, which stops right at the Space Center.

The JAXA's Tsukuba Space Center (TKSC) offers tours of its facility and exhibitions. 7 In the Space Dome you can look at many full-scale satellite models, real rocket engines, and a life-size model of a Japanese module of the ISS. You can take a picture in a space suit, too.

I bought some souvenirs in the Space Shop, and returned to Tokio to have dinner with Agustín at Tonkatsu Hamakatsu in Shinjuku. It was spectacular, as the rest of the Japanese food I had.

Sayonara Mata ne, Tokyo

I had a hard time trying to sleep on the last night, and my snoring room mates wouldn't help.

I had planned to catch the Tokyo Metro to Tokyo Station with the few remaining yens on my Suica, then catch the Keisei Bus Airport Express back to Narita. I slept in, and in the end I catched the more expensive (¥3000) Airport Limousine Bus from Shinjuku Sta., arriving on time for the check-in and board.

To be fair, I didn't expect all the transitions to be so smooth (I have an history). I had a really short timespan for one (1 hour, and 15 minutes) and was afraid to be left behind. And then, it happened:

I handled the passport and boarding pass to the airline hostess on the Seoul transition, and at the time of reading the QR code a red light buzzed.

No.

Excuse me sir, is this your last name?

No, no, no, no.

Yes.

Actually, I had previously joked about the fact that my last name appeared slightly different in the boarding pass. After a short pause, the magic words:

Free business upgrade, welcome aboard Korean Air.

A purse, slippers, the New York Times, and I actually did what Casey Neistat would have done: I changed clothes in the restroom.

Hashtag lifegoals.

Summing up, my impression of Tokyo is that it's how major cities will look in the future. Despite the fact that I didn't know most of the Japanese culture in advance, I found it amusing (unsettling, sometimes). Travelling solo has been outstanding, but it's true the experience has been very different from my previous travels as a backpacker, where it used to be very ghetto.

So, when do we open a CARTO office in Tokyo?

Here's a list of resources I used for the article besides my own notes:

My album on Flickr
Solo in Tokyo, by NYT.
Siete días en Tokio, by Alberto Romero.
Tokyo: Monocle Travel Guide I found in the hostel library.

Again, I'm so grateful to have such amazing friends who gave me some tips: Luisico (and Ju), Fran, Javier, and Agustín (which deserves a special mention himself). 8

In general, I didn't find Tokyo to be an expensive city. ↩
Take into account US plugs can be used with Japanese power sockets, but voltage is different! ↩
Shoutout: they are looking for talented visual designers. ↩
Or in English, Memory Lane (aka Piss Alley). ↩
Rice balls filled with tuna and mayonnaise. ↩
Hanami ("flower viewing") is the Japanese traditional custom of enjoying the transient beauty of flowers. ↩
If you'd like to take the tour in English you have to call in advance. ↩
I've been extremely lazy prepping the details for this trip, to the extent I realized the day before I had booked a bed in a shared female room (I called to the hostel and the amazing staff fixed it). Thus, I didn't bought a travel guide. ↩

Analyzing federal surveillance planes with CartoDB

2016-04-13T00:00:00+00:00

One week ago BuzzFeed News broke the story of U.S. government surveillance planes routinely circling over most major cities. It was while reading the Hacker News thread on the topic that I found the data, along with notes on the analysis, available on GitHub.

As I showed on my flights routes post we are working at CartoDB on a visualization solution to explore and analyze location data.

This time I didn't have to collect the data as BuzzFeed News had already done the job. I just recreated their analysis with a set of widgets for dynamic filtering.

What is #putohelicoptero looking for?

Checking the methodology used by BuzzFeed News I asked myself: "What if we could do the same analysis with Spanish cities?"

Actually, this is something many Madrid citizens have already brought up before: If you live near Madrid city center, you'd have noticed the annoying noise of a helicopter flying over the city on some quiet nights. Just to give a rough idea, there's a Twitter account called @putohelicoptero featuring the helicopter.

Digging for information I found the model of the helicopter in a news article by El Mundo, reporting all the inconveniences that the now known EC-135 'Cóndor' caused.

As explained in the BuzzFeed News analysis, Flightradar24 receives data from a network of ground-based receivers, 1 you just need an identifier for each aircraft to access that data. If only there was an aircraft registry similar to the Federal Aviation Administration registration database I could identify the model the "Policia Nacional" flights.

And it exists.

The Registro de Matrícula de Aeronaves Civiles publishes a document with all the aircraft registered in Spain. 2 At first glance, at least 41 of those aircraft were a match with the model I was looking for, and then a colleague helped me introducing them in Flightradar24 for further filtering. 3

Once we found it, we just had to export the location data and map it in CartoDB. As an example, this is the flight of the helicopter for New Year's Eve, checking Santiago Bernabéu stadium (Start of San Silvestre Vallecana), Vallecas (End of San Silvestre Vallecana), and Plaza de Sol.

At a lower scale, "Policia Nacional" is also conducting surveillance flights that can be tracked, and later analyzed. At last we can already answer the question: "What is #putohelicoptero looking for?"

We can see in which zones were conducted most of the surveillance flights, performing a spatial join to the boundaries of Madrid barrios (activate the layer in the map above). 4

Disclaimer: Flightradar24 has a pretty restrictive license, thus I haven't shared the data from #putohelicoptero. The data for federal surveillance flights is already available in BuzzFeed News GitHub account.

You can build your own receiver with a Raspberry Pi and a low cost DVB-T USB stick, or apply for a receiver ↩
Last updated on 1st April 2016, as for the day of publication of this post ↩
You need a Premium account for more than 7 days historic data, but it's easy to obtain. Thanks to @javitonino who assisted me in this process. ↩
Data from November 2015 to March 2016. ↩

Visualizing 1M flight routes with CartoDB

2016-02-23T00:00:00+00:00

Last week a friend asked on Twitter how to extract some data from Airline On-Time Performance Reports in a Tableau public dashboard. I replied back with some tricks using Tabula, when the raw data or a workbook aren't provided, and turned in.

While the data was interesting, it lacked location reference, thus eliminating the possibility of representing it on a map, not to mention further geospatial analysis. But the idea of visualizing it lived on.

Getting the data

Friday night, destroyed after a hard week. I got home, had some dinner, and started poking at FlightStats Developer Center. I grabbed a RedBull to stay awake, and wrote a script to get the rating for every route in a dataset from OpenFlights.org.

I validated the visualization with a sample of flights departing from Barcelona, with a very simple line visualization: a color ramp corresponding to a 5 star scale (red for 1, green for 5).

Several hours, a couple of "Authorization failed. usage limits are exceeded", and different API keys later I had the data.

Explore and analyze

Fast forward to this week. We've been working on some new and shiny stuff at CartoDB that has just been presented at MWC — CartoDB for Deep Insights. I had the data ready, so I uploaded it to CartoDB, and created my first Deep Insights dashboard within a couple of minutes. BOOM.

A quick look at the visualization and I can start uncovering stories.

We're continuously working to improve this, so any feedback is welcomed. Go ahead and play with the data.

You can check the code in GitHub.

Disclaimer: I'm not a Data Scientist, just some guy who knows how to put together a script in Ruby and happens to work at CartoDB. I really, really, really got inspired by this post to share the visualization. I usually tinker a lot but never end up sharing anything because I'm too critical with the results, so please be kind.

Vietnam

2015-10-28T00:00:00+00:00

Last Summer I took a holiday trip to Vietnam with the single purpose of crossing the country on a motorbike with my friend Adrián, who convinced me to join this adventure.

I had never ridden a motorbike before. Seriously, not even in high school. But after watching Top Gear: Vietnam Special — just look for it in TPB — it sounded like a great vacation plan. I had already been to Sri Lanka three years before and the idea of coming back to Southeast Asia for a long trip resonated well.

We asked for holidays at work, bought our plane tickets and waited until the last minute to prepare all the rest of stuff while drinking beers and eating croquetas. 1

We were departing from Madrid to Ho Chi Minh City, make our way to Hanoi, and then come back. The initial plan, as in the original Top Gear route, was to drive across the whole country from South to North, but after checking the distances and all we wanted to do — plus, remember the fact that I didn't know how to ride a motorbike — we discarded this option early, and ended up making use of different means of transportation: car, plane, motorbike, bike, train, bus, and boat.

It could be tempting to squeeze a lot of places to see and things to do, but we arranged a good mix of touristing, resting and traveling. We were on holidays, so the last thing you want to do is stressing out.

Regarding the motorbike: you don't need a driving license. You will find a lot of info online about how you are not supposed to ride a motorbike in Vietnam unless you have a Vietnamese driving license, besides how dangerous Vietnamese roads are. Honestly, I hardly saw any police, the roads are even better than many secondary roads in Spain and the only thing you have to worry about is to wear a helmet and orientate properly. 2

Of course this is my personal experience, and maybe we were just lucky, but I wouldn't make such a big deal of it. Either way, buy good travel insurance which covers driving a vehicle: World Nomads is the quality standard.

In the following text I basically will leave out all the general information you can find in the Lonely Planet travel guide and recommend the places and advice that worked for us. 3

Days 1 to 3Ho Chi Minh City

Ho Chi Minh City received us with a warm welcome, and after booking an Uber 4 from the Airport to the Bich Duyen hotel the first drama bursted out: I had lost my passport. I assumed this could happen at some point, but we had not even spent an hour in the country and we were already in trouble. Retracing our steps: we went to the airline desks to clarify some doubts for the upcoming internal flights (more on this later), we bought the SIM card for our smartphones 5, and we withdrew several millions of VND from the ATM. 6 I must had it forgotten somewhere along that path (with the rest of my documentation). That's where I learnt one of the good things of traveling with a sidekick: you always have a backup when things go awry. Before I could freak out the Lost and Found office of the airport contacted me to pick my passport up: phew!

A tough touristic day was awaiting the next day, so we had a Phở for dinner at Pho Quynh, went back to the hotel and called it a day.

One of the things I liked most about Vietnam was its assorted food dishes. You can have Phở for breakfast but the majority of the hotels will serve you a mix of scrambled eggs, bacon, and fruit. So that's what we had, as well as drank some juice and the worst coffee ever. Damn, as good the food is there, they still brew a horrible coffee, but we needed the energy boost.

Sputnik statue at Central Post Office

We headed to the Reunification Palace, and then we visited War Remnants Museum. You may think you know what happened in Vietnam as you've seen it in the movies and such, but words can't describe the atrocities that these poor people suffered, it was a really moving visit. We also went to see Phuoc An Hoi Quan Pagoda and we walked our way back to Backpackers district through old Saigon: Ho Chi Minh Square, Notre Dame (yes, they also have a Notre Dame), and Central Post Office which I discovered it has a Sputnik commemorative statue (I couldn't find this in any guide!).

Once at the Backpackers district, we drank the cheapest Saigon beer at some corner bar, had the tastiest Tikka Masala for dinner at Baba's Kitchen, and ordered Jack Daniels shots at a live music place before passing out.

The day after we had quite a hangover but we were just a communist burger with fries away to fix this. Luckily there was Chuck's Burgers to help and once we were fully recovered we visited the Fine Arts Museum. As said, there were several days of not doing very much, I read a lot on those days (I finished Tim Urban's SpaceX series, The Martian, and Elon Musk's biography) or just rested: it's even better at a French café drinking an iced coffee.

At the end of the day we packed up our bags and prepared everything for the next days on the road.

Days 4 and 5Hoi An

We had bought plane tickets from HCMC to Da Nang. This was one of the things I was least sure. First we were going to go by motorbike, then by train, and finally we decided to stay one night more at the hotel and fly. We wanted to be fresh and have a whole day at our disposal to get used to the motorbikes. In the end it was the best decision, whenever you can, just fly long distances.

We had already arranged with MotorVina Motorbike Rentals we would be arriving in the morning to Da Nang to pick up the bikes so we had the entire day to go to Hoi An, where we were going to be staying for a couple of days. I can still remember the face of the guy when he first saw me:

Has your friend ever ridden a motorbike?

Ha! A couple of minutes and I had already tamed the beast in our way to the closest petrol station, next stop: Hoi An. There wasn't really much to do in Da Nang so we went by the coastline until we found the village of Hoi An and we checked-in at the Full House Homestay. I was totally blown away by the kindness of the host family, who treated us like kings since the very moment we crossed the door. This wasn't an exception as this story already happened at HCMC and will repeat in Hue.

The thing to do in Hoi An is to get a bespoke shirt or dress; I also took the opportunity to walk through the market stalls, and buy some souvenirs. The food was delicious too, and we had a lunch composed of the local dishes: banh vac, banh bao, cao lau and my favourite: banh xeo. We had to wait until the tailor finished the clothes so we sunbathed in Cua Dai Beach for the rest of the day.

Day 6Hai Van Pass

The great day arrived. I mean, the whole travel made sense (if it made any) because we wanted to drive through the roads that one day served as the highway for the country. The itinerary for the day was: start at Hoi An and arrive to Hue, making some pauses in the middle.

My motorbike was a Honda 150cc

Marble Mountains
Hai Van Pass
Lang Co Bay
Elephant Springs

I really liked the sensation of driving the bike and the turns in the pass; the sights were also astonishing. Everything went smoothly until the last part of the route when it started pouring. I thought it was ok to hit pedal to the metal, this way we could arrive as soon as possible to our hotel and dry ourselves, but when Adrián overtook me to pull-over it was clear it was not ok. I still laugh when I remember telling this to our new friends some days after.

Day 7 and 8Hue

The Jade Hotel in Hue didn't let down. Actually this would be an understatement. We arrived to the entrance with the motorbikes and two bellboys parked them and brought our backpacks inside, where two fruit juices were waiting for us. Many restaurants and bars piled in the streets beside the river, where we had a well deserved dinner.

Tomb of Tu Duc, photograph by Adrián Perez

We wanted to make the most of time with the bikes so we went to Tomb of Tu Duc in the morning, and spent the afternoon in The Imperial City after waving goodbye to our beloved Hondas. Some beers at the Why not? because why not? and back to the hotel.

Tip: unlike in Spain, if you ask for peanuts you'll probably have to pay them.

The last day at Hue was really quiet as the only agenda was taking the train to Hanoi. We bought some supplies as we were going to be spending more than 12 hours (all night long) in a soft berth.

Despite we initially wanted to take the Livitrans Express (the luxury train) they were sold out and we finally took the Reunification Express. We weren't sure about the conditions of the train but it really surprised me: Adrián, who is 1.90 meters tall slept through the night.

We arrived on time to have a proper full breakfast at Puku before embarking in our Ha Long Bay love cruise.

Days 9 to 11Ha Long Bay

Maybe the most difficult part about going on a cruise by Ha Long Bay is deciding which travel agency to trust, as prices vary for the same type of cruise. Or even worse: you're sold a quality which doesn't correspond for the price you're paying. We had already done our homework and chose Lily's Travel Agency. We went on a 3-day cruise sleeping the first night at the Fantasea boat and the second night in Cat Ba island, all-but-drinks-included.

Peter —Mr. "hello, everysbody"—, our excellent guide for the next three days picked us up at Lily's office and we got on the bus for a dull 4-hour journey to the tourist boat wharf. Passengers on board, plenty of food for lunch, and first stop at a cave with a cock shaped rock. Some swimming and kayaking before dropping anchor between the beautiful karst rocks.

Ha Long Bay, photograph by Carlos Matallín

In all these travels there's this sort of epiphany where induced by the drinks and the gorgeous landscapes I think of how lucky I am; to be able to travel to the other side of the world —literally— and enjoy with some new friends we made on the boat.

That or we were drunk as hell. 7

The second day we set course to Cat Ba island, the largest of the archipelago which at its heart lies a National Park. We did some intense sweaty hiking and checked-in at the Cat Ba Palace Hotel for lunch. A quick nap after we were already on our way to Monkey Island, quite a place to celebrate the 25th birthday of the game laying at the beach. Spoiler: it's full of monkeys and no trace of Guybrush Threepwood.

After the unexpected party the day before on the boat, a couple of drinks were more than enough to return to the hotel early.

The third day on the boat was mostly to return to land first and then to Hanoi in the damn bus. But before that we attended a Vietnamese food cooking class on the deck, where we prepared our own lunch.

Days 12 and 13Hanoi

We didn't like Hanoi as much as we did HCMC, maybe because we had spent some more time in Saigon and knew it better. You can visit all the interesting places in just one day, however, you can't miss some of the best restaurants and cafés of all Vietnam.

Sharing all that time on the Fantasea cruise invited making a special bond with some people, so once we got off the bus we had a quick shower at the hotel 8 and went to have a beer and some dinner at Nola Café, the most charming café in town hands down.

The day after we woke up early to go to the Ho Chi Minh mausoleum complex, not that we were very into the Government buildings, but seeing the bearded guy everywhere got me intrigued at least. The place looks very Soviet-era as expected, and there was an special exhibit for the 70th anniversary of Vietnam People's Army. 9

The highlight of the day was the lunch at Koto, we shared a mix of starters, the duck dish, and finished with a piece of cake: delicious.

Other spots that are worth paying the visit are the Temple of Literature and the Hoa Lo Prison Museum, with the later being as impressive as the War Remnants Museum.

The trip was coming to an end and we had to catch a plane the next day, 10 so after wandering a bit through the Old Quarter streets we found a nice place to have dinner.

Day 14Back to Ho Chi Minh City

We spent the last day in HCMC visiting those places we didn't have time on the first days: Museum of Ho Chi Minh City and the saddest zoo and botanical garden in the world, but at least we enjoyed a Japanese-inspired pizza for lunch at 4P's and a fulfilling barbeque at Quan Nuong to wrap up our 2-week travel to Vietnam.

End of travel

One of the things that surprised me most about the country was despite the fact that it is a Communist regime the day to day doesn't differ that much from any other Western country. The Vietnamese way of being ranges from the uncanny kindness to the ruthless unenthusiasm, depending on the opinion of the tourists having an impact on their jobs or businesses. In any case, it is praiseworthy how they have overcome any kind of resentment the war may have provoked in them.

To cut it short: this is the perfect trip for couples —at this point I won't avoid the joke— and there's a wide range of things to do if you like outdoors, motorbikes, and food. If I missed anything it was a bit more of adventure. It's difficult to get away from the tourist circuit, but this doesn't mean you won't enjoy it a lot. Also, it's moderately cheap, so you won't spend a fortune. The only downside was it gets pretty hot depending on the hour of the day.

We're already thinking of the next destination.

Here's a list of resources I used for the article besides my own notes:

I couldn't be grateful enough to Fran, Ricardo, and Javier for proofreading an early version of this article, to Adrián for putting up with me the whole travel, and to NYT from where I got the inspiration for this post.

Note for future self: buying plane tickets with a bar public wifi may not be the safest Internet practice. ↩
This is the part where I can't enter the country anymore. ↩
The Lonely Planet travel guide is a must, to know what to see in the major cities we relied on TripAdvisor, and for the reservations of the flights and hotels we used Kayak and Booking respectively. For foreign travel advice the page of the Embassy of Spain in Hanoi is the way to go, as well as the page of the Vietnam Embassy in Spain. This post by Mar Barbera is also priceless, and helped us prepare most of our itinerary, as well as the Vietnam series of Los Junys. ↩
I'm not the biggest fan of Uber practices, but I prefered not to get ripped off by taxi drivers. It payed off because when we had a problem with a driver who got lost in one of the journeys, the company responded very well to my feedback and got a reasonable discount. ↩
Look for the MobiFone stands just by the airport exit. You can give your mobile to the girls in the desk to set it up (and a backdoor) for you. You can get a data plan for few hundred thousand VND and there's good 3G coverage in the whole country. ↩
1 Million VND ~= 40 EUR. We had enough with 6M each, and payed the food and accomodation with credit card (they charge 3% per transaction). ↩
We spent 1,000,000 VND on beers. One. Fucking. Million. ↩
By far the Backpackers Hotel was the worst of all in which we stayed, bad hygiene (cockroaches in the bathtub), not enough breakfast, and bad service (we had to ask for toilet paper). I'm not even linking it. ↩
I'm not sure if this was normal but there was propaganda on every corner. It's true that our visit happened at the same time as the country's celebration on 18 Aug. and 2 Sep. ↩
The airport is located far from the city but it only takes less than 2 hours and several thousand VND to go by bus.↩
There are several gaps (i.e. in the train arriving to Hanoi) after running out of battery. ↩