Lambros Petrou RSS Feed https://www.lambrospetrou.com/ Stay updated with all the new articles from lambrospetrou.com. Tue, 19 Aug 2025 00:00:00 GMT https://validator.w3.org/feed/docs/rss2.html https://github.com/jpmonette/feed en All rights reserved 2026, Lambros Petrou <![CDATA[Amazon DynamoDB references]]> https://www.lambrospetrou.com/articles/dynamodb/ dynamodb Tue, 19 Aug 2025 00:00:00 GMT Amazon DynamoDB is one of my favourite databases.

It’s a pain during the early stages of a project because you have to design your data model based on your queries, which is a good exercise to do anyway. If you are working on a new project without clear query patterns it can be hard to come up with an efficient data model upfront.

My favourite feature of DynamoDB is its predictability and constant performance. I know what to expect from it, and I can plan around it. Constraints are good for performance and reliability.

This article lists resources diving into how DynamoDB works under the hood.

AWS docs

Publications

Abstract: NoSQL cloud database services, like Amazon DynamoDB, are popular for their simple key-value operations, unbounded scalability and predictable low-latency. Atomic transactions, while popular in relational databases, carry the specter of complexity and low performance, especially when used for workloads with high contention. Transactions often have been viewed as inherently incompatible with NoSQL stores, and the few commercial services that combine both come with limitations. This talk examines the tension between transactions and non-relational databases, and it recounts my journey of adding transactions to DynamoDB. I conclude that atomic transactions with full ACID properties can be supported without unduly compromising on performance, availability, self-management, or scalability.

Abstract: Amazon DynamoDB is a NoSQL cloud database service that provides consistent performance at any scale. Hundreds of thousands of customers rely on DynamoDB for its fundamental properties: consistent performance, availability, durability, and a fully managed serverless experience. In 2021, during the 66 hour Amazon Prime Day shopping event, Amazon systems including Alexa, the Amazon.com sites, and Amazon fulfillment centers, made trillions of API calls to DynamoDB, peaking at 89.2 million requests per second, while experiencing high availability with single-digit millisecond performance. Since its launch in 2012, DynamoDB’s design and implementation have evolved in response to our experiences operating it. The system has successfully dealt with issues related to fairness, traffic imbalance across partitions, monitoring, and automated system operations without impacting availability or performance. Reliability is essential, as even the slightest disruption can significantly impact customers. This paper presents our experience operating DynamoDB at a massive scale and how the architecture continues to evolve to meet the ever-increasing demands of customer workloads.

Abstract: NoSQL cloud database services are popular for their simple key-value operations, high availability, high scalability, and predictable performance. These characteristics are generally considered to be at odds with support for transactions that permit atomic and serializable updates to partitioned data. This paper explains how transactions were added to Amazon DynamoDB using a timestamp ordering protocol while exploiting the semantics of a key-value store to achieve low latency for both transactional and non-transactional operations. The results of experiments against a production implementation demonstrate that distributed transactions with full ACID properties can be supported without compromising on performance, availability, or scale.

Videos and talks

Search the AWS Events channel: https://www.youtube.com/@AWSEventsChannel/search?query=dynamodb

Other

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Investing in 3x Daily Leveraged Nasdaq 100 ETFs (TQQQ or QQQ3) using QQQ weekly MACD signals]]> https://www.lambrospetrou.com/articles/investing-leveraged-qqq-macd/ investing-leveraged-qqq-macd Sun, 17 Aug 2025 00:00:00 GMT For the past few months I have been backtesting a long-term investing strategy with a friend, and we started following it since 2 months ago.

The backtesting results are great. More than +10,000% total profit from 2012 to 2025. Hopefully the future results will be too.😉💸

MACD-weekly strategy executing QQQ3 using Nasdaq 100 index (NDX) signals.

All backtesting is done in TradingView (referral link so we both get $15) using their awesome PineScript platform and the strategy tester. I love it.

Table of contents

TLDR

The strategy executes trades on 3x daily leveraged Nasdaq 100 ETFs (TQQQ in the US, and QQQ3 in the UK) using price signals from the original non-leveraged QQQ.

The strategy exploits, and depends on, the massive bull runs of the Nasdaq 100 index (see Nasdaq 100 components). Using 3x daily leveraged ETFs makes the bull runs bigger and allows the strategy to generate amazing gains on the way up, since QQQ gaining a 1% means the 3x leveraged gain is about 3%.

However, using 3x daily leveraged ETFs also magnifies the losses on the way down, so the strategy employs strict risk management in order to cap and minimize the losses while retaining the bulk of the profits.

For the rest of the article we will focus on QQQ3 which is the 3x Daily Leveraged Nasdaq 100 instrument available for trading in Europe. For folks trading in the US check out TQQQ.

OK, but what is the strategy? …

Well, you have to read the article for the details.

If you just care for the results, jump to the backtesting section.🚀

Assumptions and goals

If any of these assumptions does not hold anymore, then the strategy won’t perform as well. You have been warned!

  1. The US tech industry will continue to outperform the market over the next 10-15 years.
  2. The new technology leaders will continue to list in the Nasdaq, otherwise the Nasdaq 100 index will lose its advantage.
  3. Historically, the Nasdaq 100 has a big bull run every few years and overall trends upwards. This allows the strategy to offset losses made in whipsaw markets (oscilating up and down without clear trend).

These are the goals of the strategy and my personal constraints.

  1. The strategy has to outperform the buy-and-hold Nasdaq 100 or S&P 500 over 10-15 years.
  2. The strategy has to have better risk management from the buy-and-hold, with less overall drawdowns.
  3. No intra-day trading. I don’t have the time to sit in front of charts during the day, so working with closing prices only.
  4. No more than 5 trades per month. The fees and the UK capital gains taxes will reduce profits.
  5. Simple and easy to follow and execute.

To repeat the most important rule, the strategy has to be simple to reason about and easy to execute. There is no point having a strategy that is hard to execute correctly.

Past research with leveraged ETFs

After experimenting for a few weeks, we found a research published in 2016 by Michael A. Gayed (then updated to cover 2020) solidifying our findings and making them more concrete.

The publication “Leverage for the Long Run - A Systematic Approach to Managing Risk and Magnifying Returns in Stocks” uses the S&P 500 (largest 500 US companies) and crossovers with moving average indicators (MA) as entry and exit signals, reducing the drawdown versus buy-and-hold. Then, goes on to use leveraged ETFs (1.5x, 2x, 3x) to magnify the bull runs (gains) while also managing the drawdowns using the 200 daily moving average.

Quoting from the publication’s abstract:

This strategy shows better absolute and risk-adjusted returns than a comparable buy and hold unleveraged strategy as well as a constant leverage strategy. The results are robust to various leverage amounts, Moving Average time periods, and across multiple economic and financial market cycles.

This is a great read with lots of backtesting going back to 1928, and explains well the drawdown reduction is achieved.

Screenshot from the publication showing the results using leveraged ETFs

The above table and chart from the publication shows the impact of taking advantage of leveraged gains and capping the losses. Superb results.

Others have also discussed similar strategies, like this analysis by Logan Kane “The Trading Strategy That Beat The S&P 500 By 16+ Percentage Points Per Year Since 1928” that cites the publication above too.

Room for improvement

Even though the above research shows great potential and gives a concrete way to outperform the market (S&P 500 in that case), we can do even better.

We implemented the 200-daily Simple Moving Average (SMA) strategy of the publication using the QQQ3 ETF (3x leveraged Nasdaq 100) with 1% buffer around entries and exits, and we noticed a few things that can be improved.

Screenshot from trading view QQQ3 200D crossover with the chart, the strategy panel, and the strategy tester

  1. There are false positive trades during whipsaw periods where there is no clear trend, like in 2016.
  2. There are big drawdowns from the peak high price to our exit points (crossunder of the 200-daily SMA), wasting gains of the bull run, like in 2020.

We managed to reduce the whipsaw trades with some tricks and tweaks, but ultimately they are inevitable when the markets go sideways.

We love markets going up.

We are OK with markets going down, since we will exit.

We hate markets going sideways, since it increases costs and losses.

Weekly signals

The first differentiating aspect of my strategy is that it uses weekly candlestick charts and only executes trades using weekly close prices.

As mentioned in the previous section, we want to avoid whipsaw periods where we enter and exit trades close to each other without making much profit, if any at all. When this happens several times, losses are more likely than gains.

Using weekly charts reduces the amount of noise we have, and it also reduces the amount of trades we can do in a year. Note that we use 40-week SMA now instead of the 200-daily SMA, same total duration.

This change reduced the total number of trades, reduced the whipsaws, but also reduced the profits too.

Cross symbol strategy

Doing any technical analysis and signal detection on QQQ3 directly leads to noise and false positives, both for entries and exits, because each movement is significant.

We concluded that in this situation it’s better to use a less volatile symbol for detecting the signals, and use the more volatile but highly correlated symbol for the trade execution.

For example, the QQQ3 is the more volatile sibling of QQQ, so we detect signals on QQQ and execute on QQQ3.

Stop losses are calculated on the traded symbol, though, for proper risk management.

Moving Average Convergence and Divergence - MACD

Once we settled on using weekly signals, the next improvement comes from addressing the big drawdowns from the peak highs.

For example, even though the COVID crash of March 2020 was short in duration, it was a significant dip. The QQQ3 drawdown from peak high down to our exit when QQQ crosses under its 40W-SMA was roughly -60%.

TradingView screenshot QQQ3 with 40W strategy showing drawdown

We did many iterations and different variations of the SMA crossover approach with OK results. Tweaks and tricks here and there and employing different sub-strategies while above the 40-weekly SMA to capture more of the bull run. More details about some of these tricks might come in a separate article.

However, we decided to switch to using the weekly MACD indicator for our entry and exit signals, with tiny bit of tweaking to make it awesome.

Quoting directly from Investopedia’s MACD definition:

Moving average convergence/divergence (MACD) is a technical indicator to help investors identify price trends, measure trend momentum, and identify entry points for buying or selling. Moving average convergence/divergence (MACD) is a trend-following momentum indicator that shows the relationship between two exponential moving averages (EMAs) of a security’s price.

In a nutshell, the MACD indicator has two lines, the MACD line and the Signal line.

The MACD line is calculated using the Exponential Moving Average (EMA) of two periods:

MACD = 12-Period EMA − 26-Period EMA

The Signal line is calculated by taking the EMA of the MACD line:

Signal = MACD 9-Period EMA

When the MACD line crosses above the Signal line it means that the price is going up, they get close when the price moves sideways, and the MACD line crosses under the Signal line when the price is going down.

The key crossover for us is not between the MACD line and the Signal line, though, it’s between the MACD line and the zero line.

When the MACD line crosses over the zero line it means that now the 12-period EMA moves higher than the 26-period EMA, which most likely means a bull trend therefore we ENTER our position. When the MACD line cross under the zero line it means that the 12-period EMA moves lower than the 26-period EMA, indicating a bear trend therefore we EXIT our position.

The fact that we use weekly charts makes the MACD crossovers usable, otherwise there would be a lot of noise, even more than the simple crossover strategy we explored earlier.

Risk management with stop losses

Every successful professional trader and investor will tell you that the most important trait a trader should have is managing risks and sizing their positions appropriately.

There is an amazing talk by Dr. David Paul The Consistently Winning Trader Psychology where he explains, among other things, how to size your trades based on their risk.

Long story short, wisdom says you shouldn’t “bet or risk” more than 1-3% of your tradeable account value in each trade.

So, if the account has 100,000 USD, each trade should not risk losing more than 1-3K. What you choose exactly depends on your risk appetite and how confident you feel about your trades, so let’s settle on a 2% bet.

Note that the 2% bet, or 2K USD, is NOT the amount we should trade, it’s the max we could lose. The total amount of the trade depends on the strategy.

In my strategy we use a hard stop loss from the entry price at 10% (or 15% depending on how volatile is the traded symbol), so the total position we should enter comes out to 20K USD.

Position = AccountValue * RiskBet% / StopLoss%
=> 100_000 * 0.02 / 0.10
=> 20_000

The Nasdaq 100 makes moves larger than 10% regularly, and since we use 3x leveraged ETFs it means that a 3.4% movement from QQQ would instantly trigger our stop loss leading to false exits.

Therefore, the strategy uses dynamic stop losses on the way up. The stop loss from the entry price is a hard rule, and nothing overrides it.

However, during a bull trend, we use a trailing stop loss from the peak highs and allow up to 30% of drawdown, extended with some volatility buffer (fixed or ATR-based). This allows the 3x leveraged movements to have enough wiggle room to dip and then go up again, which is crucial to avoid false exits.

Concretely, the stop loss calculation is something like this:

stopLossPct = 10
stopLossDynamicPct = 30
bufferPct = 2

stopLossEntryPrice = entry_price * (1-stopLossPct/100)
dynamicStopLoss = highestClose * (1-stopLossDynamicPct/100) * (1-bufferPct/100)
activeStopLoss = math.max(stopLossEntryPrice, dynamicStopLoss)

shouldExitStopLoss = signalClose < activeStopLoss

Experienced folks will say that using leveraged ETFs is dangerous since they can move more than 30% in a single week.

Indeed, in theory this can happen. There is an optional escape hatch for emergency exits that we monitor manually. If the daily close price is lower than the calculated stop loss, then we can exit our position.

Better to lose some profits in case it was a false exit that recovers the next day, instead of not being able to sleep at night. We can always re-enter the position if it’s going up (see below).

Re-entry after a stop loss trigger

There are two main rules for an entering a position:

  1. MACD line crosses over the zero line.
  2. MACD line crosses over the Signal line, used only when the MACD line is above the zero line.

The first rule is used after we exit because of a bear market where the MACD line crossed under the zero line.

The second rule is used after we exited due to a stop loss while the MACD line is still above the zero line. This can happen when there are big drops but the market recovers before crossing under the zero line.

The benefit of the MACD-based strategy over the SMA-based strategy is that eventually the MACD line always crosses the MACD Signal line, even if they stay above the zero line. Prices do not always go up, at some point they will stabilise and the MACD line converges with the Signal line, leading to a crossover.

This allows us to use stop losses and protect ourselves.

Tricks and tweaks

The strategy also incorporates a few tweaks and tricks that become possible due to the fact we use different symbols for signal and execution, further optimizing our entries and exits to avoid false positives.

As we will see later in the backtesting section, this same strategy can be used with company stocks as well, like AVGO (Broadcom) or SMT (Scottish Mortgage Trust). The main requirement is that these stocks are very highly correlated with the signal symbol, in this case Nasdaq 100 (which is configurable too).

We won’t go into details for these tweaks but just to give a glimpse:

  • Relative strength. When entering a position the target symbol (QQQ3, AVGO) has to be trending upwards faster than the signal symbol (QQQ).
  • Rising or Falling. When entering or exiting a position the target symbol must be in the appropriate trend as well for N consecutive weeks. For example, enter only once there are 2 rising bars.
  • Buffer around the exit level to avoid false exits and whipsaws (sudden move down, then immediately up).
  • For the MACD Signal line calculation we use a 5-period EMA instead of 9-period EMA. Falls into the overfitting category, but it does give a slight edge to some trades recovering faster from dip drawdowns, without any drawback. You can use the 9-period EMA, though, and the overall results will be similar.
  • … few more.

All of the above are configurable and optional.

Keep in mind that these tricks can be incorporated in any strategy, even the 200-daily/40-weekly SMA crossover strategy. We did actually implement most of these for that strategy too, but the MACD-based strategy is still overall better.

Backtesting

OK, let’s see some numbers!

  • All charts below are logarithmic scale, so that the past dips are not miniscule.
  • Click each image to see it in full size.

Chart legend

The screenshots below have a panel for the active strategy right below the chart itself.

The strategy panel has certain icons that is worth knowing what they mean:

  • Green triangle: All conditions are met for an entry.
  • Red triangle: All conditions are met for an exit.
  • Red flag above red triangle: Exit triggerred because of a stop loss.
  • White cross overlay the red triangle: The trade ended up losing money.
  • Yellow triangle: MACD line crosses below the exit level, but we are waiting for the confirmation filters.

QQQ3 with QQQ signal

  • Dates: 2012-12-01 to 2025-07-31
  • Configuration: BufferPct=2%
  • Results: +10,981% PROFIT

Screenshot from trading view QQQ3/QQQ with the chart, the strategy panel, and the strategy tester

And the individual trades.

Screenshot from trading view QQQ3/QQQ with the list of trades

And this is how the 40-week SMA crossover strategy would behave when using NDX as the signal symbol (with the 2% exit buffer too). +2,800% PROFIT

Screenshot from trading view QQQ3 40-week SMA crossover

Using the QQQ3 as signal with the 40-week SMA crossover strategy returns +2,300% PROFIT.

QQQ3 with NDX signal

  • Dates: 2012-12-01 to 2025-07-31
  • Configuration: BufferPct=2%
  • Results: +12,698% PROFIT

NDX is the Nasdaq 100 index itself, whereas QQQ is an ETF tracking the index. As you see the performance is almost identical to the one using QQQ as signal, just a single trade in 2016 moves a bit earlier when using NDX as the signal symbol leading to a bit of extra profit.

Screenshot from trading view QQQ3/NDX with the chart, the strategy panel, and the strategy tester

These small differences are insignificant at the grand scheme of things, and focusing too much on these just leads to overfitting the strategy which can lead to bad results in the future.

TQQQ with QQQ signal

TQQQ is the US version of QQQ3, and the strategy works nicely as well.

  • Dates: 2010-02-08 to 2025-07-31
  • Configuration: BufferPct=2%, ConsecutiveEnterConditions=2
  • Results: +11,194% PROFIT

Screenshot from trading view TQQQ/QQQ with the chart, the strategy panel, and the strategy tester

And the individual trades.

Screenshot from trading view TQQQ/QQQ with the list of trades

EQQQ with NDX signal

EQQQ is the UCITS compliant version of QQQ tradeable in Europe. This comparison is mostly to show the behavior during market crashes, since EQQQ is in theory tracking NDX, although currency fluctuations can have an impact here too.

  • Dates: 2005-08-01 to 2025-07-31
  • Configuration: BufferPct=2%, RelativeStrengthBars=1
  • Results: +1,100% PROFIT

Notice how the strategy handles the drawdowns of 2008-2009 and 2022 very nicely. In the end, the strategy underperformed slightly the buy and hold from 2005, but the whole benefit of using this strategy is that the drawdowns we experienced were almost nothing and the single losing trade in 2008 was just a -11%.

This is why I like this strategy. Avoid holding the bag at the bottom, but capturing the meat of the bull. OK, it got too buzzwordy.

Screenshot from trading view EQQQ/NDX with the chart, the strategy panel, and the strategy tester

The purple chart (3rd top to bottom) shows the relative strength of EQQQ against the Nasdaq 100 Index (NDX). When the line is purple EQQQ is weaker than NDX, therefore it does not enter a position (see the green triangles until mid-2001), and when the line is blue EQQQ is stronger than NDX and positions can be entered.

Scottish Mortgage Trust (SMT) with QQQ signal

Scottish Mortgage Trust is a growth focused trust in the UK, which I usually invest in for global growth exposure.

Here we also use the relative strength filter as entry confirmation, since occasionally QQQ will be trending upwards while SMT downwards, and that is not a good time to enter.

In this case, we can see the strategy working throughout the crashes of the past 25 years, including the dotcom bubble in 2000, the crash of 2008, COVID crash in 2020, and the longer crash in 2022.

  • Dates: 1995-07-24 to 2025-07-31
  • Configuration: BufferPct=2%, RelativeStrengthBars=1
  • Results: +1,305% PROFIT and we avoided most of the big dips.

Screenshot from trading view SMT/NDX with the chart, the strategy panel, and the strategy tester

The purple chart (3rd top to bottom) shows the relative strength of SMT against the Nasdaq 100 Index (NDX). When the line is purple SMT is weaker, therefore it does not enter a position (see the green triangles until mid-2001), and when the line is blue SMT is stronger than NDX and positions can be entered.

AVGO with QQQ signal

  • Dates: 2009-08-03 to 2025-07-31
  • Configuration: BufferPct=2%
  • Results: +6,000% PROFIT (Note the 100% win rate!)

Screenshot from trading view AVGO/QQQ with the chart, the strategy panel, and the strategy tester

And with NDX instead of QQQ! +6,159% PROFIT

Although, to be fair, it was quite hard to lose money holding AVGO in the past 15 years. The stock just goes up. Even with the buy and hold strategy AVGO only had a max of 35-40% drawdown occasionally, so if you could stomach those you would win big.

The strategy’s goal though is to not have to stomach more than what we want, and still try to capture much of the profit.

UPRO with SPX

  • Dates: 2010-06-01 to 2025-07-31
  • Configuration: BufferPct=2%, EntryStopLoss=15%, DynamicStopLoss=30%
  • Results: +1,700% PROFIT

UPRO is the 3x daily leveraged ETF of the S&P 500 index, and SPX is the S&P 500 Index.

Screenshot from trading view UPRO/SPX with the chart, the strategy panel, and the strategy tester

The 40-week SMA crossover strategy with the 2% exit buffer would return +1,100% PROFIT and when using the SPX as signal +2,000% PROFIT.

Screenshot from trading view UPRO/SPY 40W-based with the chart, the strategy panel, and the strategy tester

Open questions

The one question we are still researching and backtesting is how to add more money into an existing position.

Everything is mechanical around entering a position and exiting a position, but considering that some periods span multiple years we need a robust way to add more money into the position even if just a few times per year, without negatively impacting our risk management.

Conclusion

This is super fun, isn’t it.😉

I suggest you read the publication “Leverage for the Long Run - A Systematic Approach to Managing Risk and Magnifying Returns in Stocks” to see the impact of having a strategy to manage risk and reduce drawdowns while also taking advantage of leverage to magnify profits.

The basis of my strategy, and any strategy using the leveraged ETFs, is that the bull markets happen regularly and they are big enough to offset the losses from bear markets when managed properly.

If you have better ideas or improvements to my strategy, feel free to reach out on @lambrospetrou or email me.

Changelog

  • 2025-11-14: Fixed the TQQQ with QQQ signal screenshot, incorrectly showing QQQ3.
  • 2025-08-19: Minor edits and new section on EQQQ vs NDX. Fixed the 200D-SMA crossover screenshot.
  • 2025-08-18: Initial public post.
  • 2025-08-17: Initial draft, not publicly listed.
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Cloudflare Durable Objects are Virtual Objects]]> https://www.lambrospetrou.com/articles/durable-objects-are-virtual-objects/ durable-objects-are-virtual-objects Sun, 27 Apr 2025 00:00:00 GMT I have been working a lot with Durable Objects as part of my day job at Cloudflare and have been answering questions on the #durable-objects Discord channel on a daily basis for several months now.

One of the most common things people have trouble wrapping their head around is how to manage and handle the lifecycle of Durable Objects.

They want to do something when a Durable Object (DO) is “created”, when it’s “destroyed”, when it’s “hibernating”, when it’s “evicted from memory”, and any variation you can imagine.

The short answer is “you do not do that”.

You do not create a Durable Object. You do not destroy a Durable Object. You just use a Durable Object.

Let’s explore in detail.

Virtual Objects or Actors

For the rest of this article, “Object” and “Actor” is used interchangeably.

Durable Objects nicely fit into the Actor programming model, built natively into the Cloudflare global network infrastructure. There are several Actor programming libraries (Akka, Microsoft Orleans) and languages (Erlang, Elixir).

My most favourite description of Virtual Actors is from the Microsoft Orleans publication “Orleans: Distributed Virtual Actors for Programmability and Scalability” back in 2014.

Actors are the basic building blocks of Orleans applications and are the units of isolation and distribution. Every actor has a unique identity, composed of its type and primary key (a 128-bit GUID). An actor encapsulates behavior and mutable state, like any object. Its state can be stored using a built-in persistence facility. Actors are isolated, that is, they do not share memory. Thus, two actors can interact only by sending messages.

Every sentence from the Orleans Actors description applies to Durable Objects as well, although Durable Object IDs are 32 bytes.

The same section continues to elaborate on the key facets of a Virtual Actor.

1. Perpetual existence

Perpetual existence: actors are purely logical entities that always exist, virtually. An actor cannot be explicitly created or destroyed and its virtual existence is unaffected by the failure of a server that executes it. Since actors always exist, they are always addressable.

Durable Objects are the same. You don’t create them. You don’t destroy them.

You generate a Durable Object ID and address the corresponding Durable Object.

2. Automatic instantiation

Automatic instantiation: Orleans’ runtime automatically creates in-memory instances of an actor called activations. At any point in time an actor may have zero or more activations. An actor will not be instantiated if there are no requests pending for it.

Durable Objects are an infrastructure primitive, so in constrast to Orleans in-memory instances, they are running on a server within Cloudflare’s global infrastructure with attached durable storage for persistence.

Similarly to Orleans though, a single Durable Object can be active and consuming resources on a Cloudflare server, or inactive and consuming zero resources.

As long as there are requests routed to the Durable Object it will be alive and active somewhere. Once requests stop and enough time passes, the Durable Object will be evicted from memory and eventually removed from its host server.

Your user code should not care about this.

When a request routes to a Durable Object, Cloudflare will prepare its durable storage and initialize the Object on a server ready to accept incoming requests. All of this happens transparently.

3. Location transparency

Location transparency: an actor may be instantiated in different locations at different times […].

Durable Objects by default are created close to the location of the incoming request. Optionally, you can provide a regional location hint to influence where the Durable Object will be created.

Having said that, the user code should not care exactly at which location the DO is created. It can be on server-a at time 10:00, and on server-b at time 22:00, where server-a is in a different city or country even from server-b.

Once again, Cloudflare will make sure the Durable Object will be running on a healthy server somewhere close to the user or within the region specified. Where that is exactly, can, and will change over time.

How to think about it

It’s understandable that many folks have trouble internalizing the above properties. The traditional way of doing things in programming languages is to “create and destruct” class objects of sorts, and at the infrastructure level we create and delete server instances.

Fully embracing the “Virtual Object or Actor” paradigm really unlocks the power of these primitives.

My guidelines are:

  • On “activation time” (Durable Object class constructor) read the storage and accordingly do any in-memory initialization needed.
  • Use explicit actions to do your business logic, and if not possible, use the Alarms API to schedule work to be done in the future.

Let’s explore some common use-cases.

Access a Durable Object

  1. Create the Durable Object ID (see docs).
  2. Get a stub to the Durable Object (see docs). Keep in mind that creating a stub does not yet attempt to reach out to the addressable Durable Object.
  3. Invoke an operation on the Durable Object stub.

The last step is what will actually start the whole flow of figuring out where the Durable Object should be activated, which server should handle the request, send the request there, run the invoked operation, and then return the response back to the caller.

Example from my Tiddlyflare project:

export async function routeListWikis(env: CfEnv, tenantId: string): Promise<ApiListWikisResponse> {
    let id: DurableObjectId = env.TENANT.idFromName(tenantId);
    let tenantStub = env.TENANT.get(id);
    return tenantStub.list();
}

Notice that all the 3 properties explained in the previous section apply.

We always just use the Durable Object regardless if it’s the first time it’s accessed or not, we don’t create it explicitly, and we don’t care about where it’s going to be instantiated.

We just use it.

Initialize state only once

Say you want to store some information in your Durable Object durable storage only once, and have that in-memory whenever the DO is active for fast access.

Example from my Tiddlyflare project:

export class WikiDO extends DurableObject {
    env: CfEnv;
    sql: SqlStorage;

    wikiId: string = '';
    tenantId: string = '';

    constructor(ctx: DurableObjectState, env: CfEnv) {
        super(ctx, env);
        this.sql = ctx.storage.sql;
        // ...
        const tableExists = this.sql.exec("SELECT name FROM sqlite_master WHERE name = 'wiki_info';").toArray().length > 0;
        if (tableExists) {
            const { tenantId, wikiId } = this.sql
                .exec<{ tenantId: string; wikiId: string }>('SELECT tenantId, wikiId FROM wiki_info LIMIT 1')
                .one();
            this.tenantId = tenantId;
            this.wikiId = wikiId;
        }
    }

In the above snippet, I check if a table exists, and if it exists I read the tenantId and wikiId values and store them in-memory.

This only covers the reading part, though, and the writing is done inside the create(...) action (see code).

In most applications, there is some specific operation (like create(...)) that receives the needed information and writes it to storage.

Delete all storage after inactivity

Another common use-case is “deleting a Durable Object” after its expected usage is over to avoid paying for storage that is not needed anymore.

If the deletion is tied to a specific operation, then you just invoke the storage.deleteAll() API on the Durable Object storage at the end of that operation and all is good.

The nuanced scenario is when the deletion of storage is tied with the “destruction” of the Durable Object, whatever that means. But, as we said Durable Objects are never deleted, so what do we do?

The solution to this is the Alarms API.

While processing requests you can set a date in the future to execute the alarm handler, and do the cleanup in the alarm handler. Every new request postpones the alarm and pushes the cleanup forward (you can use in-memory debouncing to avoid bursts of writes if that’s a concern).

Conclusion

Durable Objects are Virtual Objects.

They are not explicitly created or destroyed.

They are always uniquely globally addressable.

They are activated and deactivated automatically within the Cloudflare global infrastructure on-demand to handle requests.

You just use them.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Large Language Model (LLM) prompting guides]]> https://www.lambrospetrou.com/articles/llm-prompting-guides/ llm-prompting-guides Sat, 26 Apr 2025 00:00:00 GMT This is a “live” collection of the best Large Language Model (LLM) prompting guides I find online that help me write better prompts and in general get the most value possible out of LLMs.

LLM providers

Anthropic

Models: https://docs.anthropic.com/en/docs/about-claude/models/all-models (e.g. Claude 3.7 Sonnet)

OpenAI

Models: https://platform.openai.com/docs/models (e.g. GPT4.1)

Google

Models: https://ai.google/get-started/our-models/ (e.g. Gemini 2.5 Pro)

ElevenLabs

Others

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[DevConnect Series talk - Cloudflare Developer Platform Overview]]> https://www.lambrospetrou.com/articles/devconnect-cloudflare-dev-platform/ devconnect-cloudflare-dev-platform Fri, 03 Jan 2025 00:00:00 GMT On 2025-Jan-03, I delivered a talk doing an overview of the Cloudflare Developer Platform in my home-city, Nicosia, Cyprus 🇨🇾.

Thanks to Parsectix, specifically Pavlos, for organising the event (LinkedIn Event), and to all the folks that joined.

It was awesome to see some familiar faces in the audience.😅

The talk covers an overview of the Cloudflare Developer platform, core services, and how to gradually integrate it with your existing infrastructure as you see fit.

📽️ Download presentation slides

Some folks asked about Cloudflare’s free tier, and how cheap the plans are compared to other platforms, raising concerns around rug-pulling in the future (not uncommon with other platforms).

Cloudflare already blogged about why and how we manage to offer the free tier, and how zero egress fee is possible, at the post Reaffirming our commitment to free.

I sent this already to some, and putting it here for future readers of this page with similar concerns.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[How to do encryption and envelope encryption with KMS in Go]]> https://www.lambrospetrou.com/articles/encryption/ encryption Sun, 15 Dec 2024 00:00:00 GMT Table of contents

In this article, I will go through the basics of symmetric data encryption, then move to envelope encryption using our own key encryption key or using a Key Management Service (KMS) from a cloud provider, and how that can be generalised to multiple levels of envelope encryption.

I will provide examples in Go code taken out from my own applications to make things concrete and understandable.

Symmetric encryption

There are millions of resources about encryption, but in this article I will focus on symmetric encryption of arbitrary data.

The goal of encryption is to convert some piece of data A (namely plaintext) into some different piece of data B (namely ciphertext) using a secret data encryption key DEK. The data B will seem like random garbage to anyone looking at it, and only the ones in possesion of key DEK can reverse the conversion back to A.

Therefore, we use encryption to protect sensitive data and stop worrying if someone inadvertedly gets access to it.

The conversion from A to B is called encryption, and the reverse is called decryption.

Symmetric encryption diagram
Symmetric encryption with a data encryption key.

For example, if we encrypt the plaintext Hello, World! with a specific algorithm and secret key, we will end up with the following ciphertext:

2YjP8xC8owzTkLrvEdHjUY2q6QWicr6n1Te0sAso5oR7KaCufSiebQadhQ82js01wRd135Q

The above seemingly random text can be stored anywhere without worrying about anyone ever figuring out that the actual plaintext data is Hello, World!.

Encryption can be used everywhere and for everything.

My favourite online resources about security and encryption are the following:

As of today, the recommended algorithms to use for symmetric encryption is 256-bit Advanced Encryption Standard (AES) in Galois Counter Mode (GCM) or XSalsa20+Poly1305 with a 256-bit data encryption key.

Symmetric encryption in Go

Let’s get into specifics on how to safely implement symmetric encryption in our applications.

I will be using Go code in this article without any third-party libraries, since the Go standard library has everything we need in the crypto package (and its sub-packages).

The function GenerateKeyBytes below generates 256-bit (32-bytes) data encryption keys that will be used throughout the whole article.

import (
    "crypto/rand"

    "golang.org/x/crypto/chacha20poly1305"
)

const KeySize = chacha20poly1305.KeySize

// GenerateKey returns a 32-byte key as per the chacha20poly1305 algorithm.
func GenerateKeyBytes() []byte {
    b := make([]byte, KeySize)
    _, err := rand.Read(b)
    if err != nil {
        panic("unexpected failure: could not generate random data")
    }
    return b
}

The following functions implement symmetric encrypting/decryption.

func EncryptBytes(key []byte, plainData []byte) ([]byte, error) {
    // We use this algorithm based on the recommendation of https://www.latacora.com/blog/2018/04/03/cryptographic-right-answers/#encrypting-data
    // Alternative could be AES-256 GCM: https://pkg.go.dev/crypto/cipher#NewGCM

    aead, err := chacha20poly1305.NewX(key)
    if err != nil {
        return nil, fmt.Errorf("failed to create encryption AEAD: %w", err)
    }

    // Select a random nonce, and leave capacity for the ciphertext.
    nonce := make([]byte, aead.NonceSize(), aead.NonceSize()+len(plainData)+aead.Overhead())
    if _, err := rand.Read(nonce); err != nil {
        return nil, fmt.Errorf("failed to encrypt: %w", err)
    }
    // Encrypt the message and append the ciphertext to the nonce.
    encryptedMsg := aead.Seal(nonce, nonce, plainData, nil)

    return encryptedMsg, nil
}

func DecryptBytes(key []byte, encryptedData []byte) ([]byte, error) {
    aead, err := chacha20poly1305.NewX(key)
    if err != nil {
        return nil, fmt.Errorf("failed to create encryption AEAD: %w", err)
    }
    if len(encryptedData) < aead.NonceSize() {
        return nil, fmt.Errorf("ciphertext is too short: %d < %d", len(encryptedData), aead.NonceSize())
    }

    // Split nonce and ciphertext.
    nonce, ciphertext := encryptedData[:aead.NonceSize()], encryptedData[aead.NonceSize():]

    // Decrypt the message and check it wasn't tampered with.
    plainData, err := aead.Open(nil, nonce, ciphertext, nil)
    if err != nil {
        return nil, fmt.Errorf("failed to decrypt: %w", err)
    }

    return plainData, nil
}

And to test the above you can use the following test:

func TestEncryptDecrypt(t *testing.T) {
    key := []byte("supersecretkey32byteslong1234567")
    plaintext := "Hello, World!"

    ciphertext, err := encryption.Encrypt(key, plaintext)
    if err != nil {
        t.Fatalf("Encrypt failed: %v", err)
    }
    if ciphertext == plaintext {
        t.Errorf("Encrypt: plaintext same as cipherText, got %s, want %s", ciphertext, plaintext)
    }

    decryptedPlaintext, err := encryption.Decrypt(key, encResult.CipherText)
    if err != nil {
        t.Fatalf("Decrypt failed: %v", err)
    }
    if decryptedPlaintext != plaintext {
        t.Errorf("Decrypt: plaintext mismatch, got %s, want %s", decryptedPlaintext, plainText)
    }
}

Data encryption keys

The data encryption key (DEK) is maybe the most crucial component in encryption (apart from the encryption algorithm itself). Anyone with the DEK can decrypt all data encrypted with that key, therefore securely storing DEKs is top priority.

You should always separate your plaintext data encryption keys from encrypted data.🔐

On one side of the security spectrum, there are applications with a single encryption key used to encrypt everything, and pass this key into the application with approaches like environment variables, something like Hashicorp Vault, or storing the key(s) in Amazon S3-compatible stores and restricting access with IAM permissions.

This is easy to manage, since you only worry about securing a single key, but it’s extremely dangerous in case you lose it or if someone gets access to it when they shouldn’t.

On the other end of the spectrum, you generate a different data encryption key for each piece of data your application wants to encrypt.

This seems much safer, since in the worst case that someone gets ahold of a key they can only decrypt a single piece of data, but management of all these keys sounds like nightmare.

Enter envelope encryption.

Envelope encryption

In the previous section, we saw that a good approach to reduce risk and blast radius in case someone gets access to a data encryption key (DEK) is to have many of them, one per piece of data. Securely storing and managing all these DEKs is a concern though.

Envelope encryption introduces another kind of key, the key encryption key (KEK). The key encryption key (KEK) is used to encrypt the data encryption keys (DEKs). Once a DEK is encrypted, it can be safely stored together with the encrypted data.

Therefore, we are back at having to manage a single key encryption key while each piece of data is encrypted with its own data encryption key.

To envelope encrypt some data:

  1. Generate a data encryption key (DEK) using our GenerateKeyBytes() function from above.
  2. Encrypt our plaintext data with the DEK from step 1, using the EncryptBytes() function.
  3. Encrypt the DEK itself using a secret key encryption key (KEK), using the EncryptBytes() function.
  4. Store the data ciphertext from step 2 and the DEK ciphertext from step 3 into our database/datastore.

To envelope decrypt some data:

  1. Decrypt the data encryption key (DEK) ciphertext first using the key encryption key (KEK), using the DecryptBytes() function from above.
  2. Decrypt the data ciphertext using the decrypted DEK from step 1, using the DecryptBytes() function.

If you have been following along, we solved the problem of managing many data encryption keys (DEKs), but we still need to securely manage the key encryption key.

I have seen many teams storing their key encryption keys in whatever “secret store” their cloud providers provide, since every single platform has a way to store secrets.

This is usually fine, assuming that your provider is properly implementing their “secrets store” with an actual Key Management Service (KMS) under the hood where their employees (or any intruder into their systems) cannot access the actual secret values.

Leaks of “secrets” from different providers is not uncommon though.

We can do better by using a Key Management Service (KMS) to securely store our key encryption keys (KEKs) and handle the encryption/decryption of our data encryption keys (DEKs). That way, the KEKs are never being transmitted in plain text, reducing the risk of them being leaked.

Enter Key Management Services, usually referred to as KMS services.

Envelope encryption with KMS

Envelope encryption allows us to use different data encryption keys (DEKs) for our data, and encrypt those DEKs with one or a few key encryption keys (KEKs).

The best and recommended way to securely store and manage those key encryption keys (KEKs) is with a dedicated key management service (KMS).

These KMS services encrypt and decrypt data encryption keys without ever exposing the actual key encryption keys in plaintext. This significantly reduces the risk of leaking our KEKs, since our application doesn’t even have access to them.

We offload trust to the KMS service provider.

Having said that, there is still risk. But, we offload that risk to the cloud provider of our choice. We trust that they implemented these KMS services correctly, and that if any leak or intrusion happens into their own systems our key encryption keys are still secure.

There are a lot of security measures taken by KMS service providers to ensure the key encryption keys are securely protected, and it usually involves keeping the plaintext version of our keys inside Hardware Security Modules (HSM) volatile memory just for the few milliseconds needed for the operation. See Data protection in AWS Key Management Service for more details.

Most reputable cloud providers offer KMS services, for example AWS KMS and Google Cloud Key Management. Many companies also use Hashicorp Vault that allows integration with various KMS services and provides secrets access across the entire infrastrure.

The main API offerred by KMS services that we care about is:

  • EncryptKey
  • DecryptKey
  • GenerateDataKeyPair

Let’s examine how we can use a KMS service for envelope encryption.

Symmetric envelope encryption with KMS service
Symmetric envelope encryption with KMS service.

To envelope encrypt some data:

  1. Generate a data encryption key (DEK) using our GenerateKeyBytes() function from above.
  2. Encrypt our plaintext data with the DEK from step 1, using the EncryptBytes() function.
  3. Encrypt the DEK itself passing the DEK to the KMS EncryptKey API.
  4. Store the data ciphertext from step 2 and the DEK ciphertext from step 3 into our database/datastore.
Symmetric envelope decryption with KMS service
Symmetric envelope decryption with KMS service.

To envelope decrypt some data:

  1. Decrypt the data encryption key (DEK) ciphertext first using the KMS DecryptKey API.
  2. Decrypt the data ciphertext using the decrypted DEK from step 1 and the DecryptBytes() function from above.
  3. Return the data plaintext from step 2.

The GenerateDataKeyPair API (see docs) can replace our own GenerateKeyBytes() function from above. Calling this API will return a data encryption key in both plaintext and ciphertext. We would use the plaintext version to encrypt our data, and store the ciphertext alongside the encrypted data so that we can pass it later to the KMS DecryptKey API when we will want to decrypt the data. The plaintext should be discarded immediately after encrypting the data.

Envelope encryption in Go

The following code is what I use to abstract away any KMS service to allow me to mock them out in local tests, or use any kind of KMS service regardless of the provider.

type KeyEncryptionWrapper interface {
    EncryptKey(ctx context.Context, keyPlain []byte) ([]byte, error)
    DecryptKey(ctx context.Context, keyCipher []byte) ([]byte, error)
}

type Enveloped struct {
    kms KeyEncryptionWrapper
}

func NewEnveloped(kms KeyEncryptionWrapper) *Enveloped {
    return &Enveloped{
        kms: kms,
    }
}

Now let’s see the encrypt/decrypt methods of the Enveloped abstraction.

// Encrypt encrypts the given data and returns Base64 formatted cipher data blob
// that needs to be passed to the `Decrypt()` function for decrypting.
func (e *Enveloped) Encrypt(ctx context.Context, data []byte) (string, error) {
    var err error
    dekKey := GenerateKeyBytes()
    dekCipher, err := e.kms.EncryptKey(ctx, dekKey)
    if err != nil {
        return "", fmt.Errorf("could not encrypt data key: %w", err)
    }

    cipherData, err := EncryptBytes(dekKey, data)
    if err != nil {
        return "", fmt.Errorf("could not encrypt data: %w", err)
    }

    return fmt.Sprintf(
        "%s.%s",
        ToBase64(dekCipher),
        ToBase64(cipherData),
    ), nil
}

// Decrypt extracts the necessary key information from the cipherDataBlob, decrypts the data
// and returns the decrypted plain bytes.
func (e *Enveloped) Decrypt(ctx context.Context, cipherDataBlob string) ([]byte, error) {
    var err error
    partsDot := strings.SplitN(cipherDataBlob, ".", 3)
    if len(partsDot) != 2 {
        return nil, fmt.Errorf("invalid cipher data blob")
    }
    dekCipher, dataCipher := fromBase64(partsDot[0]), fromBase64(partsDot[1])
    if dekCipher == nil || dataCipher == nil {
        return nil, fmt.Errorf("invalid cipher data blob")
    }

    dekKey, err := e.kms.DecryptKey(ctx, dekCipher)
    if err != nil {
        return nil, fmt.Errorf("could not decrypt data encryption key: %w", err)
    }

    plainData, err := DecryptBytes(
        dekKey,
        dataCipher,
    )
    if err != nil {
        return nil, fmt.Errorf("could not decrypt data: %w", err)
    }

    return plainData, nil
}

Note that I use a Base64 encoded string as the encryption result to make it more readable and universally compatible with any storage product, and it also makes the decrypting a bit simpler. That’s an optional step, and alternatively I could just return two byte slices directly, one for the DEK ciphertext and one for the data ciphertext.

Key rotation

We haven’t discussed anything about key rotation so far, but it’s a key component in keeping your data safe.

Key rotation is the re-encryption of data with a different data encryption key (DEK).

  1. Retrieve the data encryption key (DEK) for some data.
  2. Decrypt the data ciphertext using the DEK from step 1.
  3. Generate a new data encryption key (DEK-2).
  4. Encrypt the data using DEK-2.
  5. Store data ciphertext from step 4.

Key rotation is important in order to reduce the risk of a key being leaked or being discovered impacting much of your encrypted data.

Even if an attacker gets ahold of a data encryption key (DEK) plaintext, they will only be able to decrypt data that used that specific DEK. Therefore, rotating the encryption keys makes this attack impossible.

Even though key rotation is nice, it becomes very hard and gets very costly if we had to re-encrypt the entirety of our dataset. Imagine Google wanting to re-encrypt the Google Drive files across all their customers, or Amazon S3, or Cloudflare R2. You get the idea. It’s not trivial or practically feasible to re-encrypt petabytes of data every day.

Envelope encryption helps here again.👌

Since we have separate DEKs for each piece of data, we can do re-encryption only for the DEK itself which is stored alongside the encrypted data. The encryption keys are a few bytes (32-bytes in our code above), therefore much more practical to re-encrypt across the board.

KMS services offer key rotation APIs to simplify the key rotation process, and you can even have multiple key encryption keys eligible for decrypting a data encryption key so that you can carry out key rotation over several days or weeks.

Multi-level envelope encryption

The vast majority of users should use envelope encryption with a KMS service and everything will work out fine.

In some cases, when you need to manage a lot of key encryption keys, you might want to apply envelope encryption multiple times.

For example, AWS uses multiple levels of hierarchical envelope encryption. They generate an account level key encryption key used to encrypt key encryption keys for each service in that account (e.g. S3). Then, that service key encryption key is used to encrypt data encryption keys to encrypt data stored by the service.

As another example, in Skybear.NET I generate one account-level key encryption key (AKEK) that is encrypted/decrypted by AWS KMS, and then that AKEK is used to encrypt/decrypt the individual data encryption keys for each piece of data encrypted for that account.

The main reason to do this approach is to reduce the amount of KMS API calls, either due to cost reasons or due to KMS API rate limits.

Keeping the intermediate key encryption keys in memory for a few seconds or minutes could be very beneficial in some cases. But, unless you really need this optimization, stay with the simple KMS-based envelope encryption and avoid complexity.

Conclusion

If you have read till here, thank you!🙏🏼

To summarize:

  • Encrypt your data, encrypt your users’ data, encrypt everything.
  • Follow security guidelines, use secure algorithms and correct implementations of those algorithms. Use Go more😉
  • Use KMS services to offload encryption key management.
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[How to detect website text content changes with Skybear.NET]]> https://www.lambrospetrou.com/articles/detect-website-text-changes/ detect-website-text-changes Sun, 08 Dec 2024 00:00:00 GMT Table of contents

Hurl is a CLI tool that makes testing and automating HTTP APIs easy and enjoyable. For the past year, I have been building a managed platform to run Hurl scripts for you (Skybear.NET), automatically scaling the underlying infrastructure and providing useful execution reports.

The following article is a copy of the Skybear.NET corresponding How-to Guide, posting it here for my records (canonical URL properly used😉), and its content is as of 2024-Dec-08.

In this article I will use Skybear.NET to continuously check a website’s text content and notify me when a specific text changes.

Real world scenario to detect PagerDuty vCard updates

As part of being oncall at work, we use PagerDuty for alerting the oncall engineers when a team gets paged.

However, the PagerDuty app has several issues depending on the device you use and the OS version regarding things like Do not Disturb mode, leading to missed page calls. What’s the point of being oncall and not getting alerted😅

One easy (and maybe dumb) solution I have been doing for a few years to always guarantee that the phone calls by PagerDuty always “make a sound” is to import the PagerDuty vCard directly into my contacts. Therefore, even if I don’t have the PagerDuty app installed, as long as I allowlist the PagerDuty contact entry to always alert regardless of silent mode, I will always get alerted.

The website PagerDuty vCard Updates has a section listing the latest version of the PagerDuty vCard Update as a date, e.g. 2024-11-13.

PagerDuty vCard Updates website

In this guide we will periodically fetch the above website, detect changes in the specific version date of the latest vCard update, and notify us in case it changes so that we can download the new vCard.

If you want to play with the final script, run it with the Open Editor. No signup required, and you can play with it for FREE.

Note that as of the time of writing this guide, the latest vCard update date is 2024-11-13.

Script to detect vCard updates

Since we will be doing HTML inspection we will be using the Hurl’s XPATH assertion capabilities. A nice cheatsheet for XPath can be found at https://devhints.io/xpath.

Let’s take a look at the HTML section we care about on the PagerDuty website:

<!-- more content -->
<h3 class="heading heading-3 header-scroll" align="">
    <div class="heading-anchor anchor waypoint" id="latest-vcard-update"></div>
    <div class="heading-text">
        <div id="section-latest-v-card-update" class="heading-anchor_backwardsCompatibility"></div>
        Latest vCard Update
    </div>
    <a
        aria-label="Skip link to Latest vCard Update"
        class="heading-anchor-icon fa fa-anchor"
        href="#latest-vcard-update"
    ></a>
</h3>
<ul>
    <li>2024-11-13</li>
</ul>
<!-- more content -->

As you see from the HTML snippet above, we will need to find the <h3> element that has a child element with the ID latest-vcard-update (the first <div> child above). Once we have the <h3> element, we will find the immediate sibling <ul>, and its text content will be the vCard latest update date we are interested in.

Let’s breakdown our XPath query:

  1. Get the <h3> element that has a child with the expected ID:
    //h3[.//*[@id='latest-vcard-update']]
  2. Get the first <ul> sibling of the <h3> element from step 1:
    //h3[.//*[@id='latest-vcard-update']]/following-sibling::ul[1]
  3. We will normalize the text content of the <ul> element and its children to remove leading and trailing whitespace simplifying our assertion (see normalize-space() docs):
    normalize-space(string( ... ))

The full XPath selector we will use is:

normalize-space(string(//h3[.//*[@id='latest-vcard-update']]/following-sibling::ul[1]))

For comparison, the corresponding JavaScript query selector would be:

document.querySelector("h3:has(#latest-vcard-update) + ul").textContent.trim();

We have done the hard part now🎉 Let’s write our Hurl script to periodically fetch the website, extract the vCard update date, and compare the vCard update date against the last date we have downloaded the vCard.

// detect-pagerduty-vcard-changes.hurl
# PagerDuty vCard updates detection
GET https://support.pagerduty.com/main/docs/notification-phone-numbers#pagerduty-vcard
HTTP 200
[Asserts]
xpath "normalize-space(string(//h3[.//*[@id='latest-vcard-update']]/following-sibling::ul[1]))" == "2024-11-13"

As of the time of writing this guide, the latest vCard update date is 2024-11-13.

The moment that PagerDuty will update their vCard, the assertion above will fail and if you have configured email notifications Skybear.NET will notify you immediately.

Below you can see an example of how the assertion failure would look like if our assertion was expecting 2024-11-10:

error: Assert failure
  --> ./s_nsFlFDlJkX54hqRSFFhGkf7-5srrVV5lz1Pq.hurl:5:0
   |
   | GET https://support.pagerduty.com/main/docs/notification-phone-numbers#pagerduty-vcard
   | ...
 5 | xpath "normalize-space(string(//h3[.//*[@id='latest-vcard-update']]/following-sibling::ul[1]))" == "2024-11-10"
   |   actual:   string <2024-11-13>
   |   expected: string <2024-11-10>
   |

Scheduled runs

Now that we have a script to monitor content changes, we can create a scheduled cron trigger to make sure it runs continuously every day and sends us an email when the content changes.

After you create the Skybear.NET script with the appropriate content, navigate to its Settings tab, and configure a Scheduled Cron trigger with the cron expression 0 1 * * * so that it runs every day at 01:00, forever.

You can configure the trigger to notify you by email when the content changes are detected.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Testing the Cloudflare D1 REST API with Hurl]]> https://www.lambrospetrou.com/articles/hurl-cloudflare-d1/ hurl-cloudflare-d1 Sun, 24 Nov 2024 00:00:00 GMT Table of contents

As I wrote in a previous post, I love using Hurl to test HTTP JSON APIs. Hurl is a command-line interface tool (CLI) that makes testing and automating HTTP APIs easy and enjoyable.

In this article, I will showcase some Hurl features with a concrete example. We will test part of the Cloudflare D1 REST API.

You can find the whole Hurl file mentioned below in this Gist: https://gist.github.com/lambrospetrou/4e07bf79abea9fd82b52d1a6f985405c

User journey

Cloudflare D1 is Cloudflare’s take on a serverless SQL database. It’s built ontop of SQLite and specifically the SQLite in Durable Objects product. Read the blog post Zero-latency SQLite storage in every Durable Object for more details.

D1 is meant to be used from within a Worker using the D1 Worker Binding API for best performance. It provides however a REST API that allows scripts and automation to interact with a database instance from anywhere.

Note: The D1 REST API is not optimized for performance, since all requests go through a central location, but can be useful for adhoc queries.

The user journey we will test is the following:

  1. Create a D1 database named skybear-test-001.
    • This can fail if a database already exists with that name.
  2. List the databases of the account and extract the database UUID for the one named skybear-test-001.
    • We use a list operation to figure out the database ID since the previous step can fail, hence we might not have the ID of the same-named existing database.
  3. If step 1 failed, it means the database existed already, so delete the database with the ID from step 2.
  4. If step 1 failed, we need to create a new database instance.
  5. Get the database details and assert its details.
  6. Submit an SQL query with one CREATE and one SELECT statements and assert the results.
  7. Submit the same queries as step 6, but requesting “raw” results instead of the default and assert the results.
  8. Cleanup the database.

Steps 1-4 could be simplified into a single step just creating the database, but I decided to do the above extra steps in the sake of showcasing some Hurl features and making our test suite more robust.

Hurl source

The Cloudflare D1 REST API requires an account API token, and the accountId (find your account ID) to be provided in all requests.

We will use Hurl variables to extract both of these values and simplify our scripts to not contain hardcoded secrets. The variables are CLOUDFLARE_TOKEN and CLOUDFLARE_ACC_ID.

Create a new database

POST https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
{
  "name": "skybear-test-001",
  "primary_location_hint": "weur"
}
HTTP *
[Captures]
db_created: jsonpath "$.success"

A simple POST request, already showcasing some Hurl niceties.

We can specify request headers (e.g. Authorization) right below the HTTP method and URL of the request.

The multiline inlined JSON between lines 3-6 will automatically set the Content-Type: application/json header to our request.

The HTTP * line indicates that we expect an HTTP response code of anything, hence the *. The reason we don’t assert the exact code is that as we explained, this create request can fail if a database already exists with the same name.

The last line is a captured variable, and is what we will use in steps 3-4. We create a new variable named db_created that will have the value of the success property of the JSON response.

A value true for db_created denotes that there wasn’t any existing database with the same name and all went well, whereas a value of false denotes that the creation failed, and we need to delete the existing database first.

Hurl supports JSONPath for easy parsing of JSON API responses.

Ensure freshness

As mentioned above, we want our main test requests to query a newly created database, so the steps described in this section delete any existing database with the same name and re-create it.

We want to find the ID of the existing database (step 2 of user journey). The following Hurl source does that by listing all databases and extracting the ID of the one named skybear-test-001.

GET https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
HTTP 200
[Captures]
db_id: jsonpath "$.result[?(@.name == 'skybear-test-001')].uuid" nth 0

Line 3 asserts that we received an HTTP response code 200 indicating success, and line 5 creates a new variable db_id that has the UUID value of the existing database.

This JSONPath documentation is nice for understanding the syntax, but it basically filters the result array for items that have name == 'skybear-test-001' and then returns the uuid field for each item selected.

The final nth 0 ensures that our variable will have a single string value and not an array value, since we will reuse it in subsequent requests.

The following snippet issues a DELETE request to the appropriate URL using the db_id variable (step 3).

DELETE https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database/{{ db_id }}
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
[Options]
skip: {{ db_created }}
HTTP 200

The interesting bit here is the skip: {{ db_created}} request option.

If the skip option value is true the request will not be sent. This ensures we only delete the database if step 1 above failed.

POST https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
[Options]
skip: {{ db_created }}
{
  "name": "skybear-test-001",
  "primary_location_hint": "weur"
}
HTTP 200
[Captures]
db_id: jsonpath "$.result.uuid"

Similarly to the deletion, in this step we create a new database (step 4), only if step 1 failed, and then we assign the newly created database ID to the same variable db_id.

We use the skip: {{ db_created}} request option again to only do this if necessary.

Database details

At this point we have a new database ready to accept our queries.

GET https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database/{{ db_id }}
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
HTTP 200
[Asserts]
jsonpath "$.success" == true
jsonpath "$.result.uuid" == {{ db_id }}
jsonpath "$.result.name" == "skybear-test-001"
jsonpath "$.result.running_in_region" == "WEUR"

We do a straightforward GET request here to assert the basic details of the database.

Using Hurl’s powerful assertions we ensure it has the name and ID we expect, and is placed in the location we provided during creation (step 1 or 4).

Database queries

We now want to execute SQL queries on our database.

The /query endpoint (see docs) accepts an sql string that can contain multiple SQLite statements and responds with an array of results, one for each statement.

In the example below, we issue two statements, one to create a new table, and one to select a few rows from the SQLite built-in table sqlite_master.

POST https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database/{{ db_id }}/query
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
Content-Type: application/json
{"sql": "CREATE TABLE IF NOT EXISTS marvel (name TEXT, power INTEGER); SELECT name, type FROM sqlite_master ORDER BY name ASC;"}
HTTP 200

[Asserts]
jsonpath "$.success" == true
jsonpath "$.result[0].success" == true
jsonpath "$.result[0].results" count == 0
jsonpath "$.result[1].success" == true
jsonpath "$.result[1].results[0].name" == "_cf_KV"
jsonpath "$.result[1].results[0].type" == "table"
jsonpath "$.result[1].results[1].name" == "marvel"
jsonpath "$.result[1].results[1].type" == "table"

# SQL Duration in the Durable Object should be FAST! (less than 2ms)
jsonpath "$.result[0].meta.duration" < 2.0
jsonpath "$.result[1].meta.duration" < 2.0

There shouldn’t be anything new in the above snippet apart from the fact that we use more Hurl filters in our assertions. For example, jsonpath "$.result[0].results" count == 0 asserting that our first result (the INSERT statement) has zero rows returned.

The last two lines assert that the SQLite statement execution duration was less than 2 milliseconds. Yes, SQLite in Durable Objects is fast🚀

The following is an excerpt of the actual response:

{
    "result": [
        {
            "results": [],
            "success": true,
            "meta": {
                "duration": 0.2732 //...
            }
        },
        {
            "results": [
                {
                    "name": "_cf_KV",
                    "type": "table"
                },
                {
                    "name": "marvel",
                    "type": "table"
                }
            ],
            "success": true,
            "meta": {
                "duration": 0.2147 //...
            }
        }
    ],
    "success": true
}

Finally, our last test will be against the /raw endpoint that is identical to the /query above but instead of returning an array of objects as results, it returns the raw rows and separately the column names (see docs).

POST https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database/{{ db_id }}/raw
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
Content-Type: application/json
{"sql": "CREATE TABLE IF NOT EXISTS marvel (name TEXT, power INTEGER); SELECT name, type FROM sqlite_master ORDER BY name ASC;"}
HTTP 200

[Asserts]
jsonpath "$.success" == true
jsonpath "$.result[0].success" == true
jsonpath "$.result[0].results.columns" count == 0
jsonpath "$.result[0].results.rows" count == 0

jsonpath "$.result[1].success" == true
jsonpath "$.result[1].results.columns[0]" == "name"
jsonpath "$.result[1].results.columns[1]" == "type"
jsonpath "$.result[1].results.rows[0][0]" == "_cf_KV"
jsonpath "$.result[1].results.rows[0][1]" == "table"
jsonpath "$.result[1].results.rows[1][0]" == "marvel"
jsonpath "$.result[1].results.rows[1][1]" == "table"

# SQL Duration in the Durable Object should be FAST! (less than 2ms)
jsonpath "$.result[0].meta.duration" < 2.0
jsonpath "$.result[1].meta.duration" < 2.0

Almost identical, with the assertions of the rows and columns being the difference.

At the very end we should cleanup by deleting the database:

DELETE https://api.cloudflare.com/client/v4/accounts/{{ CLOUDFLARE_ACC_ID }}/d1/database/{{ db_id }}
Authorization: Bearer {{ CLOUDFLARE_TOKEN }}
HTTP 200

OK, that’s it.

With a few lines we have tested and asserted almost all of the D1 REST API.

Skybear.NET

Finally, a piece of self-promotion😅

If you use Hurl for HTTP API testing, and have scripts you wish you could run on a schedule or on-demand as part of your CI pipeline I am building Skybear.NET doing exactly that.

The platform provides you comprehensive reports for every single script run execution that you can view at any time. The full HTTP response headers and bodies are automatically persisted for you, for every execution, which makes investigations and troubleshooting of your APIs easy and simple.

The full script we examined so far is running as-is on the Skybear.NET platform as we speak, configured to run every few minutes. Your scripts can execute on the platform without any changes from how you run them locally.

Try Skybear.NET and send me your feature requests.

Conclusion

If you are doing anything with HTTP-based APIs and websites, do yourself a favour and integrate Hurl into your daily workflow.

Hurl is amazing, and comes with a CLI that runs all your test files in parallel by default, generates detailed JSON and HTML reports, and it’s overall a great way to ensure correctness of your APIs.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Control and data plane architectural pattern for Durable Objects - Cloudflare Reference Architecture Diagram]]> https://www.lambrospetrou.com/articles/durable-objects-control-data-plane-pattern/ durable-objects-control-data-plane-pattern Wed, 20 Nov 2024 00:00:00 GMT This is a reference architecture diagram entry I originally published on the Cloudflare Documentation website.

This article is a copy of the original doc above for my records 😅 Don’t worry, all credits are given to the documentation page above with a corresponding canonical URL.

Introduction

Durable Objects are built on-top of Cloudflare Workers spanning several locations across our global infrastructure network. Each Durable Object instance has its own durable storage persisted across requests, in-memory state, single-threaded execution, and can be placed in a specific region.

A single Durable Object instance has certain performance and storage capabilities. Therefore, to scale an application without being restricted by the limits of a single instance we need to shard our application data as much as possible, and take advantage of the Cloudflare infrastructure by spreading our Durable Object instances across the world, moving both the data and compute as close to the users as possible.

This document describes a useful architectural pattern to separate the control plane from the data plane of your application to achieve great performance and reliability without compromising on functionality.

  • The control plane provides the administrative APIs used to manage resource metadata. For example, a user creating and deleting a wiki, or listing all wikis of a user.
  • The data plane provides the primary function of the application and handles the operations on the resources data directly. For example, fetching and updating the content of a wiki, or updating the content of a collaborative document. Data planes are intentionally less complicated and usually handle a much larger volume of requests.
  • The management plane is an optional component of a system providing a higher level of interaction than the control plane to simplify configuration and operations. In this document, we will not focus on this as the same principles apply as to the control plane.

Control and data plane separation pattern

In this pattern, our application consists of at least one Durable Object instance per resource type handling all its control plane operations, and as many Durable Object instances as we need for the data plane operations, one for each resource instance created in the application.

You can scale to millions of Durable Object instances, one for each of your resources.

The main advantage of this architectural pattern is that our data plane operations, usually with larger volume of requests than control plane operations, are handled directly by the Durable Object instances holding the resource data without going through the control plane Durable Object instance. Therefore, the application’s performance and availability is not limited by a single Durable Object instance, but is shared across thousands or millions of Durable Objects.

Consider an example for a generic resource type XYZ, where XYZ could in-practice be a wiki, a collaborative document, a database for each user, or any other resource type in your application.

Figure 1: Control and data plane architectural pattern for Durable Objects

  1. A user in London (LHR) initiates a resource XYZ creation request. The request is routed to the nearest Cloudflare datacenter and received by the Workers fleet which serves the application API.
  2. The Worker code will route the request to the appropriate control plane Durable Object instance managing the resources of type XYZ. We will use the idFromName approach to reference the Durable Object instance by name (control-plane-xyz). This allows immediate access to the control plane Durable Object instances without needing to maintain a mapping.
    • The location of the control plane Durable Object will be close to the first request accessing it, or to the explicit region we provide using Location Hints.
  3. The control plane Durable Object instance (control-plane-xyz) receives the request, and immediately creates another Durable Object instance (data-plane-xyz-03) near the user request’s location (using Location Hints) so that the actual Durable Object instance holding the resource’s content is near the user that created it.
    • We call a custom init(...) function on the created Durable Object instance (data-plane-xyz-03) passing any required metadata info that will be needed to start handling user requests. The Durable Object instance stores this information in its local storage and performs any necessary initialisation. This step can be skipped if each subsequent request to the created resource contains all the information needed to handle the request. For example, if the request URL contains all the information as path and query parameters.
    • We use the idFromName approach to reference the Durable Object (data-plane-xyz-03) which allows the use of name-based resource identifiers.
    • Alternatively, we can use the newUniqueId approach to reference the Durable Object which will give us a random resource identifier to use instead of a name-based one. This random identifier will need to be communicated back to the user so that they provide it in their subsequent requests when accessing the resource.
  4. The control plane Durable Object instance (control-plane-xyz) stores the generated identifier (data-plane-xyz-03) to its local storage, in order to be able to list/delete all created resources, and then returns it to the Worker.
  5. The user receives a successful response for the creation of the resource and the corresponding identifier, and (optionally) gets redirected to the resource itself.
  6. The user sends a write request to the API for the resource identifier returned in the previous step, in order to update the content of the resource.
  7. The Worker code uses the resource identifier provided to directly reference the data plane Durable Object instance for that resource (data-plane-xyz-03). The Durable Object instance will handle the request appropriately by writing the content to its local durable persistent storage and return a response accordingly.
  8. Another user from Portland (PDX) is sending a read request to a previously created resource (data-plane-xyz-01).
  9. The Worker code directly references the Durable Object instance holding the data for the given resource identifier (data-plane-xyz-01), and the Durable Object instance will return its content by reading its local storage.

As long as the application data model allows sharding at the resource level, you can scale out as much as you want, while taking advantage of data locality near the user that accesses that resource.

The same pattern can be applied as many times as necessary to achieve the performance required.

For example, depending on our load, we could further shard our control plane Durable Object into several Durable Objects. Instead of having a single Durable Object instance for all resources of type XYZ, we could have one for each region. The name-based approach to reference a Durable Object instance simplifies targeting the appropriate instance accordingly.

In conclusion, as long as you find a way to shard your application’s data model in fine-grained resources that are self-contained, you are able to dedicate at least one Durable Object instance to each resource and scale out.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Love letter to Hurl]]> https://www.lambrospetrou.com/articles/hurl/ hurl Sun, 03 Nov 2024 00:00:00 GMT Table of contents

Hurl is a command-line interface tool (CLI) that makes testing and automating HTTP APIs easy and enjoyable.

Yeap, “testing”, “easy”, and “enjoyable”, in the same sentence. Not common.👌

I realized that I have been posting a lot about the managed platform I am building to run Hurl scripts for you (Skybear.NET), without writing about the underlying star of the show.

This article elaborates on why I use and love Hurl for most of my end-to-end testing and HTTP automation needs. From testing Go server APIs, to JavaScript APIs on Cloudflare Workers, to simply chaining a few API calls together to integrate different services into a pipeline.

Hurl

Hurl is a command line tool that runs HTTP requests defined in a simple plain text format.

It can chain requests, capture values and evaluate queries on headers and body response. Hurl is very versatile: it can be used for both fetching data and testing HTTP sessions.

Hurl makes it easy to work with HTML content, REST / SOAP / GraphQL APIs, or any other XML / JSON based APIs.

— By hurl.dev

The hurl.dev website is really, really good. Detailed documentation with plenty examples and snippets of how to do anything you want with Hurl (see Samples docs).

The above quote is how they introduce Hurl. Notice that they don’t focus on “just testing”.

Hurl’s power is that it’s an HTTP request automation tool. Testing is just one use-case of it.

The fact that you can chain requests, extract data from responses and pass them down to subsequent requests, and target any API content type makes it an amazing Swiss-army knife tool for anything that speaks HTTP.

It’s curl wrapped in a nice package! ❤️

Love letter

I have been writing code for more than 15 years at this point. Professionally (getting paid for it) for more than a decade.

I tried a lot of testing frameworks and tools, some with success, some with frustration, and some that were rejected on first sight. There are hundreds if not thousands of testing tools, from language-specific frameworks, to generic tools, to in-house hand-written tests with code.

I first found out about Hurl 2 years ago, when it was still at version 4.0.0 (see all releases).

Hurl was a love at first sight kind of thing.😍 Not only that, but each of the releases since, make it even more awesome.

GET https://www.skybear.net/_live-demo/secure.json
Authentication: Bearer sample-token-123
HTTP 200
{"ok":true}

Look at the above snippet. Zero unnecessary overhead or boilerplate.

I would bet that you already understand what it does just by looking at it, but let’s go through it.

Line 1 sends a GET request. Line 2 configures the Authentication header of the request.

Line 3 asserts that we get back a 200 response status code. Line 4 asserts the full response body content.

There is nothing to remove to make it simpler. That’s freakin awesome!

There is a more flexible and more powerful [Asserts] block with special helpers to do assertions. For example, the above snippet is equivalent to the following:

GET https://www.skybear.net/_live-demo/secure.json
Authentication: Bearer sample-token-123
HTTP 200

[Asserts]
body == "{\"ok\":true}"
# or
jsonpath "$.ok" == true

Are you not interested in assertions, and just want to fire off some requests? Have at it:

GET https://www.skybear.net/_live-demo/secure.json
Authentication: Bearer sample-token-123

Do you want to repeat a request 5 times with 100ms space between them?

GET https://www.skybear.net/_live-demo/get.json
[Options]
repeat: 5
delay: 100ms

Run the above examples for free with the Skybear.NET Open Editor.

Hurl has a plethora of assertions you can do (see Asserting Response docs).

It also has cool variable capturing from responses (see Capturing Response docs) which is useful for extracting data from a response body or header and reusing it later either in assertions or subsequent requests. This is very useful for example when you are creating a new resource by issuing a POST request and extracting the created resource’s ID from the response in order to query other endpoints for that identifier next.

Specifying the body for the request is as simple as the URL and headers too. You can send JSON, XML, GraphQL, just a multiline string, multipart form-data, and everything else you need (see Request docs).

Another feature I love for easy testing is that you can use injected variables in your scripts (see Injecting Variables docs).

Imagine for example that your API has a staging deployment at staging.example.com and the production deployment at www.example.com. You can adapt your actual Hurl file to use https://{{subdomain}}.example.com and inject the subdomain variable when you run the CLI.

Variables can be injected as CLI arguments (--variable subdomain=staging), as environment variables (HURL_subdomain=staging hurl <rest cli arguments>), or put all of them in a staging.env file and pass the file as command line argument (--variables-file staging.env).

Personally, I usually start off with the CLI arguments, and then once the scripts are fully fleshed out move them to .env files.

Hurl also comes with other goodies.🥳

Running all the *.hurl files in a directory tree in parallel makes up for a super fast testing experience, limited only by the target API’s latency! It’s written in Rust after all, and taking full advantage of it.

My favourite goody feature is the --report-json option (introduced in 5.0.0 - see announcement) which creates a detailed JSON report of every request made by Hurl, while also dumping every single response body received and referencing it in the report. This is a great way to debug my APIs actually end-to-end, with full introspection to headers, duration timings, and full response bodies!🚀

I only covered the tip of the iceberg. It’s so good!

Conclusion

If you are doing anything with HTTP-based APIs and websites, do yourself a favour and integrate Hurl into your daily workflow.

Make your HTTP tasks easier, simpler, and more enjoyable.

I use Hurl in my personal projects for quick and easy end-to-end tests without wasting time with unit tests and code refactoring during feature development (see Tiddlyflare example). I use it at work for testing and different automation scripts or experimentation I do with different internal and external APIs. I use it for my automation needs integrating a bunch of APIs together into a pipeline.

Finally, a piece of self-promotion 😅, if you use Hurl, and have scripts you wish you could run on a schedule or on-demand as part of your CI pipeline, with comprehensive reports you can view at any time including storing the full responses, have a look into Skybear.NET, and let me know what you like and don’t like.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Deploy your applications on a server with zero downtime]]> https://www.lambrospetrou.com/articles/server-deploy-scripts/ server-deploy-scripts Mon, 28 Oct 2024 00:00:00 GMT Table of contents

In this guide I will explain the deployment scripts I have been using for several years now for my own applications deployed on actual servers, either VPS on Hetzner and Linode or cloud instances like AWS EC2.

All the scripts below are also available at https://gist.github.com/lambrospetrou/aaaa13344f0026d810700f1bd2601cfd.

Context

Let’s clarify what we want to achieve.

We want to deploy on servers. Not serverless platforms, not managed container platforms. We have our application artifact, maybe a Go/Rust binary, maybe a zip/tar/jar file, a Python/Node script, and want to run it on the server.

We want zero downtime. This means we can deploy the application without failing in-progress requests or incoming requests while the application is being deployed.

This is important since applications I deploy on my servers are usually SQLite-based and therefore all requests are routed onto that single server.

Lastly, we want simple deploy scripts that can run from a CI like Github Actions or AWS CodeBuild and from laptops/machines locally without changes needed.

ℹ️ Note 1: Even though this guide is not for Docker containers, the Caddy section and the zero downtime section actually do apply to Docker containerized applications too.

ℹ️ Note 2: I love the approach explained in this article, because it doesn’t depend on language-specific libraries to achieve zero downtime, as long as your application supports graceful shutdowns (more in Zero downtime deployments section).

Directory structure

I like structure in my filesystem. Inspired from past teams I worked in, articles I read, and from my own needs, this is what I now use across my servers for my applications.

The directory /opt/apps_workspace/<application-name> is the “application root directory”.

Within the application root directory I have a versions/ subdirectory that contains all versions of my application deployed (I maintain the N latest versions only, more on that later).

Depending on the application, the contents of the versions/ directory could just be a list of binaries, or if my application needs multiple files it will be a list of subdirectories, one per version.

Next to the versions/ directory, there is the current/ directory that contains anything that is only relevant for the running application, e.g. .env.local, SQLite database files, and anything else not tied to a specific version but needed at runtime.

There is a symlink current/<appname> pointing to the corresponding binary file (or whatever artifact) under the versions/ subdirectory.

Live example from my Skybear.NET staging server:

$ ll /opt/apps_workspace/monosource-server/**
/opt/apps_workspace/monosource-server/current:
total 12K
drwxr-xr-x 3 appuser appadmins 4.0K Jul 25 23:03 appdata
-rw-r--r-- 1 lambros lambros   1.9K Sep 12 07:51 .env.local
lrwxrwxrwx 1 root    root       224 Sep 12 07:51 monosource-server -> /opt/apps_workspace/monosource-server/versions/monosource-server-20240912_075143-commit_unknown-f6145af482657e71162e0b105b2429fa754a0a5e11cb8d4ebf1ae6220c832a859e8d6d3f9c79fae35fd8ddb929fdb5300e34587bf2035908ef8be17638584fd2

/opt/apps_workspace/monosource-server/versions:
total 690M
-rwxr-xr-x 1 lambros lambros 69M Sep  9 21:57 monosource-server-20240909_215719-commit_unknown-1c0f46816dac2d4c9989b6ae5dbeacadfc95d26fb1d1b3c3476972b73e366a1f70f7857218b1fe8d6cd47a2ca17516193f9311f463e84cfa089e3aa992c3ff0e
-rwxr-xr-x 1 lambros lambros 69M Sep 10 07:23 monosource-server-20240910_072327-commit_unknown-dfe8db0911da484fc9f68c783765752d10b6aa923166829ddc96dc2da89a740134bca2eb30baca2c90b40a11c4ccfc8a1edcd75ca794c19abe27f18640824c42
-rwxr-xr-x 1 lambros lambros 69M Sep 10 07:40 monosource-server-20240910_074005-commit_unknown-3f4165508c7a54c27d43c20cb1417ac276f3e3c882b99bf4ac6f5a24a89283dc0af6185680802516cfb7f00767d68f441be46ff56297ba2296888432eb653fda
-rwxr-xr-x 1 lambros lambros 69M Sep 10 07:49 monosource-server-20240910_074934-commit_unknown-8b987287bfff734ea67d0877d4fe3233bc7f7f02d160437d5e97fc710603c167dafcaf656161f073b6f0cd502031566d9ccf80817a3493ee50f55b6e6617f515
-rwxr-xr-x 1 lambros lambros 69M Sep 10 07:49 monosource-server-20240910_074949-commit_unknown-8b987287bfff734ea67d0877d4fe3233bc7f7f02d160437d5e97fc710603c167dafcaf656161f073b6f0cd502031566d9ccf80817a3493ee50f55b6e6617f515
-rwxr-xr-x 1 lambros lambros 69M Sep 11 00:31 monosource-server-20240911_003106-commit_unknown-4aea354fc94a9137fbffa788892ecfe5dfb54ce2fdbd89b7ca4db15bf2a644aafacb1bfa24f9f858fdcb9108ebede6cbf5b8580afb827985fdc4bb75bae9f4b4
-rwxr-xr-x 1 lambros lambros 70M Sep 11 21:27 monosource-server-20240911_212745-commit_unknown-df30c74a6392ec0d404d77bb0f11512d457d77eba62d26316e1587aee7b1410eed23852cb4b03983f9cc19f6a5f9db8eb464f41dd5d5e5449c3a9913c78c56eb
-rwxr-xr-x 1 lambros lambros 70M Sep 11 22:26 monosource-server-20240911_222551-commit_unknown-496ff2e89f5b13190d2965a7861abdc6924adf3c45e54a5a367a27118703bcff033b402d9e758e3cbe68833e11be60acc7e3efe1fe8e9c27597d020c5b90e52b
-rwxr-xr-x 1 lambros lambros 70M Sep 11 22:38 monosource-server-20240911_223751-commit_unknown-ebd934d5e765b0651a8a045e47d4017365129abded1b02ef8fa8370b909db79fb28e9c2a7591b0adf3eaad6a6cd23f4ba1ea688eccde577ee8375afa6cdef034
-rwxr-xr-x 1 lambros lambros 70M Sep 12 07:51 monosource-server-20240912_075143-commit_unknown-f6145af482657e71162e0b105b2429fa754a0a5e11cb8d4ebf1ae6220c832a859e8d6d3f9c79fae35fd8ddb929fdb5300e34587bf2035908ef8be17638584fd2

Systemd

Systemd (https://systemd.io) is the defacto tool to manage processes, e.g. restarting them when they crash, startup on boot, and a LOT more.

We are going to use systemd to automatically start our applications on server reboots and on application crashes. We also use its Journal feature, essentially a component for capturing logs from the application’s process standard IO (stdout, stderr).

Each application I deploy has the following service file /etc/systemd/system/<appname>.service:

[Unit]
Description=Monosource Server
After=network.target

[Service]
ExecStart=/opt/apps_workspace/monosource-server/current/monosource-server
User=appuser
Group=appadmins
WorkingDirectory=/opt/apps_workspace/monosource-server/current/
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Notice that the main executable to run is our symlink in the current/ subdirectory as explained in the previous section.

Also, the app is running under the user appuser and the group appadmins, just to have some isolation from my own system user.

We specify that we want to always restart the application, and that we want both the standard output and standard error to be logged into systemd’s Journal.

To tail the application logs:

sudo journalctl -u monosource-server -f

We are going to see in the Deploy script section when this file is created, updated, and how we notify systemd to pickup changes.

Caddy

Caddy server (https://caddyserver.com) is an amazing proxy server written in Go.

Very good performance, with a user-friendly configuration syntax. We are going to use its reverse_proxy feature (see docs) for proxying all the applications on the server, and its load balancing feature to achieve zero downtime (see docs).

We are going to use the Caddyfile syntax to configure Caddy.

The main configuration file is in /etc/caddy/Caddyfile, and this is its content:

import sites-enabled/*

Yes, a single line importing all other Caddyfiles from the directory /etc/caddy/sites-enabled/.

Each application will have its own /etc/caddy/sites-enabled/<appname>-Caddyfile configuration.

This is the configuration I have for Skybear.NET which is an application serving multiple domains:

:80 {
    reverse_proxy http://127.0.0.1:8080 {
        header_up X-Real-IP {remote}

        # This gives 10s of buffering to allow zero downtime restarts of the service.
        lb_try_duration 10s
    }
    request_body {
        max_size 1M
    }
}

The above config specifies that Caddy server will listen on port :80 and act as a reverse proxy forwarding all requests to a process running locally and listening on http://127.0.0.1:8080 for HTTP requests.

I use Cloudflare as a proxy in front of all my servers, hence I don’t need to serve HTTPS from my origin servers (therefore no need to listen on port :443). If you want to go straight to your servers without any proxy CDN, Caddy supports HTTPS out of the box (see docs), so you just need to add some extra configuration for the domain(s) to certify.

In the forwarded request the headers X-Real-IP will be set appropriately, and the maximum request body we accept is 1MB otherwise the request will be rejected. Read through https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#defaults to see if you really need these headers depending on your CDN used (if any at all).

We are going to see in the Deploy script section when this file is created, updated, and how we notify Caddy to pickup changes.

Let’s now explore the lb_try_duration 10s configuration.

Zero downtime deployments

ℹ️ Assumption: Your application supports graceful shutdowns. In order for zero downtime to work properly your application has to support graceful shutdown. That means you do not abruptly kill your process during restarts, but instead process in-flight requests without accepting new ones, and then exiting the process. How you do this depends on the language and framework you use. For example here is how it’s done in Go servers: https://pkg.go.dev/net/http#example-Server.Shutdown

Without the line lb_try_duration 10s, everything would still work correctly.

The behavior would be that all requests are received by Caddy, forwarded to the local server, and the server writes its response back.

During deployments, our application will restart, hence any request being forwarded to http://127.0.0.1:8080 will fail since no server is listening on that port until the application process starts up again.

My applications are usually written in Go, so downtime is only 1-2 second(s) max. Imagine though that you use something slower (e.g. Python, Ruby) or you have to do a slow initialization in your application. That would lead to downtime of your service, which is not good.

That’s where the magic of lb_try_duration 10s comes into play.

With that line we instruct Caddy to keep retrying to reach the “backend” (our application) up to 10 seconds before failing the request.

This is awesome, since it allows our application to restart, do its initialization, and then go online to start serving the “pending” requests.

One line. Really nice 👌

Deploy script

Now that we explored all individual components of our application, let’s see the connecting tissue. The deployment script.

The following deploy script will do the following in order:

  1. Generate the application version name based on commit (if it exists) and current time (see example of this in the Systemd section).
  2. Copy all the files the application needs into a temporary directory.
    • This step ensures that all the files are copied on the target server before trying to restart any component to avoid any partial deployment.
  3. We send a big shell command over SSH (or Tailscale SSH) that will do the following:
    1. Move all the application files from the temporary directory into the /opt/apps_workspace/<appname>/versions/ directory accordingly.
    2. Move the Systemd configuration from the temporary directory to /etc/systemd/system/<appname>.service, and trigger its daemon to reload its configuration.
    3. Move the Caddy configuration from the temporary directory to /etc/caddy/sites-enabled/<appname>-Caddyfile, and trigger the Caddy daemon to reload its configuration.
    4. Update the symlink /opt/apps_workspace/<appname>/current/<appname> to point to the new version.
    5. Restart the application using systemctl restart <appname>.
      • This is the only part that causes downtime, but is mitigated by using Caddy’s load balancing feature to buffer requests.
  4. Delete older versions to retain only the latest 10 on the server, just in case I need to rollback to a previous version.
    • Rollback is done manually now by recreating the current version symlink to point to a previous version and restarting the application.

Deploy script:

#!/usr/bin/env bash

set -e

if [ "$#" -ne 1 ]; then
    echo "usage: $0 user@server-address"
    exit 1
fi

SERVER_SSH=$1
SERVER_PATH=/opt/apps_workspace/monosource-server
BINARY_NAME="monosource-server"
SERVER_RESTART_COMMAND="systemctl restart $BINARY_NAME"
SYSTEMD_FILE="monosource-server.service"
SYSTEMD_DAEMONRELOAD_COMMAND="systemctl daemon-reload"

# https://caddyserver.com/docs/running#unit-files
CADDY_RESTART_COMMAND="systemctl reload caddy"
CADDYFILE="monosource-server-Caddyfile"

# Assume the script will be run inside the `src/` directory.
OUTFILE="./build/$BINARY_NAME"
ENVFILENAME=".env.local"
ENVFILE="./build/$ENVFILENAME"
# COMMIT_HASH=$(git rev-parse HEAD)
COMMIT_HASH="commit_unknown"
BUILD_TIMESTAMP=$(TZ=UTC date -u +"%Y%m%d_%H%M%S")
FILE_HASH=$(b2sum $OUTFILE | cut -f1 -d' ')
REMOTE_FILENAME="$BINARY_NAME-$BUILD_TIMESTAMP-$COMMIT_HASH-$FILE_HASH"

echo "Deploying: $REMOTE_FILENAME"

# Copy necessary files from current version.
scp "$OUTFILE" "$SERVER_SSH:/tmp/$REMOTE_FILENAME"
scp "$ENVFILE" "$SERVER_SSH:/tmp/$REMOTE_FILENAME-$ENVFILENAME"
scp "_tools/files/etc/caddy/sites-enabled/$CADDYFILE" "$SERVER_SSH:/tmp/$REMOTE_FILENAME-$CADDYFILE"
scp "_tools/files/etc/systemd/system/$SYSTEMD_FILE" "$SERVER_SSH:/tmp/$REMOTE_FILENAME-$SYSTEMD_FILE"

# Put the latest files in the right directories and restart everything without downtime.
ssh -q -Tt $SERVER_SSH <<EOL
    sudo nohup sh -c "\
    mkdir -p $SERVER_PATH/versions/ $SERVER_PATH/current/ /etc/caddy/sites-enabled/ && \
    mv "/tmp/$REMOTE_FILENAME-$CADDYFILE" "/etc/caddy/sites-enabled/$CADDYFILE" && \
    $CADDY_RESTART_COMMAND && \
    mv "/tmp/$REMOTE_FILENAME-$SYSTEMD_FILE" "/etc/systemd/system/$SYSTEMD_FILE" && \
    $SYSTEMD_DAEMONRELOAD_COMMAND && \
    mv "/tmp/$REMOTE_FILENAME-$ENVFILENAME" "$SERVER_PATH/current/$ENVFILENAME" && \
    mv "/tmp/$REMOTE_FILENAME" "$SERVER_PATH/versions/$REMOTE_FILENAME" && \
    chmod +x "$SERVER_PATH/versions/$REMOTE_FILENAME" && \
    rm -f "$SERVER_PATH/current/$BINARY_NAME" && \
    ln -s "$SERVER_PATH/versions/$REMOTE_FILENAME" "$SERVER_PATH/current/$BINARY_NAME" && \
    $SERVER_RESTART_COMMAND"
EOL

echo "Deleting older versions, retaining the latest 10!"

# Cleanup old versions, and retain the last 10 deployed.
# In order to retain 10x versions we need to keep the top 10 lines when
# sorted with the latest files at the top, and start removing from line 11!
# Attention: If you have less than 10 deployments already this will fail, but it's fine to ignore.
ssh -q -Tt $SERVER_SSH <<EOL
    sudo nohup sh -c "find "$SERVER_PATH/versions/" -type f -exec realpath {} \; | sort -r | tail -n +11 | sudo xargs rm"
EOL

Conclusion

I love serverless platforms (you should use Cloudflare Workers and Durable Objects more), but I also love writing small servers in Go and use SQLite 😅

This article describes the deployment script I use to achieve zero downtime deployments across actual servers, VPS, or cloud instances like AWS EC2.

Feel free to copy and modify them to your will, or reach out with questions and/or ways to improve them.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Building a global TiddlyWiki hosting platform with Cloudflare Durable Objects and Workers — Tiddlyflare]]> https://www.lambrospetrou.com/articles/tiddlyflare/ tiddlyflare Tue, 22 Oct 2024 00:00:00 GMT Table of contents

Tiddlyflare is an open source TiddlyWiki hosting platform built with Cloudflare’s SQLite Durable Objects and Workers.

It supports multiple users, each with their own collection of TiddlyWikis, and each user’s data is automatically placed in a Cloudflare region close to them. Each TiddlyWiki hosted by Tiddlyflare can be up to 1GB (limits will be raised to 10GB soon).

This article goes into the architecture behind Tiddlyflare, and showcases the power and flexibility you get from Durable Objects (DO), and the Workers platform overall, without you really doing much more work.

You can find the actual source code for Tiddlyflare implementing everything described in this article at https://github.com/lambrospetrou/tiddlyflare. It works.

Context

Let’s introduce some useful background context.

TiddlyWiki

TiddlyWiki is a unique non-linear notebook for capturing, organising and sharing complex information — tiddlywiki.com

TiddlyWiki is an amazing open-source tool created more than a decade ago by Jeremy Ruston, and as the quote above mentions can be used for note-taking, information and knowledge organization, and much more.

For the purposes of this article, you only need to know that by default each wiki stores all of its data in a single HTML file. 😅

When you load that HTML file into a browser you get a UI, and the data is embedded into the HTML file itself.

There are a million ways to persist your changes (see Saving wiki section for supported plugins), but in this article we will focus on the PutSaver saving mode.

Automatically saving changes with PutSaver

We will build a platform to host TiddlyWikis. While editing our wikis we want the changes to be automatically saved remotely on the platform.

Conveniently, TiddlyWiki has a very basic but super flexible API interface named PutSaver that we will implement (see PutSaver code).

PutSaver is very simple. Once the TiddlyWiki HTML file is loaded in a browser, it sends an OPTIONS request to the current URL location, and based on the response it decides if PutSaver is supported.

If the response contains a header named dav with any value, and a success response status code (200 >= status < 300), then we are good to go.

From that point on, any changes you make to your wiki are automatically propagated with a PUT request to the current URL location and the request body is the whole HTML file. Not only the changes, but the whole file!

We don’t care if this is efficient or not, that’s how PutSaver works, and that’s what we will use.

Durable Objects

I already wrote an article introducing Durable Objects a few weeks ago, so keeping this short.

Durable Objects (DO) are built ontop of Cloudflare Workers (edge compute). Each DO instance has its own durable storage persisted across requests, in-memory state, executes in single-threaded fashion, and you decide its location. You can create millions of them!

I said it before, and saying it again. Durable Objects is the most underrated compute and storage product by Cloudflare.

It’s so different than other platforms that it’s not easy to realize its power initially. Once you get it though, it’s a game changer! 🤯

Requirements

Let’s get started with the requirements of our platform.

  1. We want to support multiple users, each with their own collection of TiddlyWikis, and the usual CRUD functionality (create/read/update/delete).
  2. Each user’s data should be isolated from each other, i.e. user A only has access to their own TiddlyWikis.
  3. Each TiddlyWiki created on the platform gets its own URL, and visiting that URL responds with the latest version of the wiki HTML file.
  4. The PutSaver saving mechanism should be supported for all wikis by default.
  5. We want all user data to be close to the user’s location. A user in Portland has their data near Portland, and a user in London has their data close to London.
  6. Support scale (intentionally leaving this open-ended to your imagination limits).

Requirement 5 is where things get hairy, really fast, with traditional hosting/cloud/infrastructure platforms.

High-level architecture

There are many approaches to implement this and satisfy the requirements. Some are complex, others are complicated, and others are just a pain in the ass.

The following design exploits and showcases the goodies of Durable Objects, satisfying all the requirements, without the code getting complex or our mental reasoning having trouble.

Tiddlyflare high level architecture diagram
Tiddlyflare high-level architecture diagram.

Let’s break down the above diagram.

  • User 1
    • First user is in Portland, US.
  • User 2
    • Second user is in London, UK.
  • Traffic to Cloudflare network (A)
    • The traffic from all users is routed to the nearest Cloudflare datacenter using Anycast.
  • Workers fleet (B)
    • The Workers are stateless, and each user request goes to any available machine inside a datacenter close to the user request’s location. Not necessarily the same machine each time.
  • Durable Object with ID TENANT_1 (C)
    • A Durable Object (DO) instance is created the first time a user attempts to create a wiki, placed in a location near the user’s request, under the Durable Object Namespace TENANT.
    • TENANT_1 is the specific DO instance created for User 1, and is placed inside a datacenter near the Portland region, close to User 1 location.
    • The tenant Durable Objects hold data about the user itself, and a metadata record for each wiki created for that user.
    • The tenant Durable Object does NOT manage any of the wiki data (i.e. the wiki HTML).
  • Durable Objects with IDs WIKI_1 and WIKI_2 (D)
    • A Durable Object (DO) instance is created when a wiki is created, placed in a location near the TENANT_1 DO location, under the namespace WIKI.
    • WIKI_1 and WIKI_2 are the specific DO instances created for User 1’s wikis.
    • We are going to understand later why the TENANT_1 location is used here instead of the User 1’s location.
  • Durable Object with ID TENANT_2 (E)
    • TENANT_2 is the specific DO instance created for User 2 information, and is placed inside a datacenter near the London region, close to User 2 location.
  • Durable Object with ID WIKI_3 (F)
    • WIKI_3 is the DO instance created for User 2’s wiki, near the location of TENANT_2 Durable Object instance.
  • Cloudflare network (N)
    • Each Worker and Durable Object instance can communicate with other instances across Cloudflare’s network without going to the public internet (most of the time).
    • This can be used to efficiently communicate between DO instances, or other Cloudflare services.

CreateWiki data flow

Let’s now explore a concrete example of how data flows through the above diagram when User 1 creates their first wiki.

  1. User 1 opens Tiddlyflare (e.g. on tiddly.lambros.dev), logs in, and clicks the button to create a TiddlyWiki.
  2. The POST request triggered flows to the nearest Workers fleet within a datacenter in Portland.
  3. The Worker code attempts to create a Durable Object Stub for User 1’s tenant ID.
    • Since this is the first time we attempt that, there is no Durable Object instance with that ID, therefore the platform will create one in the closest datacenter with Durable Object support, often in the same region.
    • The Worker code doesn’t need to check anything to see if a TENANT DO already exists and worry about all that. When you create a reference to the DO you want to use, if it doesn’t already exist it gets created, and then you just get routed to it.
    • Worker code to get access a DO instance:
      const doStub = env.TENANT.idFromName(tenantId).get();
    • The above line will return a Durable Object Stub which allows us to call methods on the DO instance directly.
    • This stub can reference a DO instance in the same datacenter, on the same machine, or in the other side of the world.
  4. Now that the TENANT_1 DO stub is created, we call doStub.createWiki(...) to create the first TiddlyWiki.
  5. The Durable Object TENANT_1 now receives the request, initiates its local SQLite storage with the appropriate SQL tables for the user information (remember; this is the first time the user did anything), and subsequently attempts to create the Durable Object that will manage the wiki’s data.
  6. Similar to step 3, we now attempt to get a stub on the wiki DO WIKI_1.
    • Generate a random ID for the wiki (i.e. WIKI_1): const doId = env.WIKI.newUniqueId();
    • Get the DO stub: const wikiStub = await env.WIKI.get(doId);
    • Create the wiki: await wikiStub.create(tenantId, wikiId, name, wikiType);
    • The WIKI_1 DO instance is created near the TENANT_1 location because that’s the origin of the request to the WIKI_1 DO instance.
  7. The WIKI_1 DO instance initializes its own local SQLite database with the right tables to store wiki data, and stores the default TiddlyWiki HTML file.
  8. The TENANT_1 DO instance receives the successful response from WIKI_1, writes in its own SQLite database that WIKI_1 was created successfully, and returns the information about the newly created wiki back to User 1.
    • The URL of a wiki includes the WIKI_1 ID, therefore just having the URL is enough to be able to reference the WIKI_1 DO instance without having to access TENANT_1 at all.
    • Alternatively, if you want to expose “names” instead of IDs through URLs, you can use the idFromName(name) function to create your DO ID (see docs).

Are you starting to see the magic? 👁️

GetWiki data flow

We have a wiki created now, so let’s see the much simpler wiki data read flow.

  1. User 1 opens the wiki URL returned by the creation flow.
  2. The GET request flows to the nearest Workers fleet within a datacenter in Portland.
  3. This time the Workers code attempts to create a Durable Object Stub straight to WIKI_1, and bypasses the TENANT_1 DO, since the URL encodes the DO instance ID.
    • Worker code to the wiki DO instance:
      const doId = extractWikiId(requestUrl)
const wikiStub = env.WIKI.idFromString(doId).get();
    • As before, WIKI_1 is probably in the same region as TENANT_1 or even same datacenter.
  4. The worker now calls the DO stub to return the wiki content.
    • Code: return wikiStub.getFileSrc(wikiId);
  5. The WIKI_1 DO instance will wake up, if not already running, read from its local SQLite database the contents of the wiki HTML file, and stream it back to the Worker.
  6. The Worker simply forwards the stream of the DO response back to User 1, without any intermediate buffering for no added overhead.

In summary, all GET requests for a wiki flow through the nearest worker to the user location, they then get routed to the corresponding WIKI Durable Object instance holding that wiki’s data (could be on the other side of the world), and the content is streamed back to the user.

That’s it.

User 2 accessing User 1 wiki

Can you guess what the read flow looks like for User 2 trying to read WIKI_1 from User 1? For simplicity let’s assume all wikis are publicly accessible to anyone with the URL at hand (in reality we have actual auth).

Go ahead and guess which step of the previous section would be different.

OK.

The differences are steps 2 and 3.

The request from User 2 will go to the Workers fleet nearby User 2’s location, somewhere in London.

The WIKI_1 Durable Object instance doesn’t move after creation, therefore in step 3 the Worker will create a WIKI_1 DO stub that will reach out to the Portland region to access the WIKI_1 DO instance, and then continue with the wiki reading as usual.

Location hints

There was a subtle difference between step 3 and step 6 in the creation flow (User 1).

In step 3, the TENANT_1 Durable Object (DO) instance is created in the nearest datacenter to the user location. Whereas, in step 6, the WIKI_1 Durable Object instance is created in the nearest datacenter to the TENANT_1 DO instance’s location.

In general, the location considered when creating a Durable Object instance is the location of the running code that attempts to create the Durable Object Stub.

In step 3, the stub creator is the Worker code running closest to the user, whereas in step 6 it’s the code running inside the TENANT_1 DO instance.

In this specific example, both Durable Object instances are going to end up in the same region since they are all close by, maybe even the same datacenter/machine, but in other scenarios this might be different. See the example where User 2 attempts to access WIKI_1.

In cases where you want to influence the location of a Durable Object instance, ignoring the location of the stub creator, you can use Location Hints and provide explicitly the region you want to place the Durable Object instance.

let durableObjectStub = OBJECT_NAMESPACE.get(id, { locationHint: "eeur" });

// Supported locations as of 2024-Oct-22
wnam    Western North America
enam    Eastern North America
sam     South America
weur    Western Europe
eeur    Eastern Europe
apac    Asia-Pacific
oc      Oceania
afr     Africa
me      Middle East

👉🏼 Tip: https://where.durableobjects.live is an amazing little website showing all the Cloudflare locations with Durable Object support.

OK. So what.

You might be wondering, apart from the fact that we used Cloudflare proprietary technology, why this, and not use a VPS, or AWS Lambda, or anything else really.

This is why I love Durable Objects and what made the Workers platform really click for me.

  1. You are not tied down to a single location. It’s TRIVIAL to put data in any of the Cloudflare regions supporting Durable Objects. Create a Stub to the Durable Object you want, in the location you want, and get to work.

    • Imagine having to write a Cloudformation stack to deploy across 10+ regions, and communicate within those regions from your application. All the configuration needed, managing all those regional endpoints. My god! 🤬
    • For anyone saying skill issue right now, I worked at AWS. I have been using AWS for almost a decade. I have implemented CI/CD pipelines to deploy across all its regions (with AWS SAM, AWS CDK, Terraform).
    • The difference is night and day. It doesn’t even come close!

  2. Scale to hundreds, thousands, hundreds of millions of Durable Object instances.

    • A Durable Object is a mini server with local, durable, actual disk storage (10GB).
    • With just a name or ID, you summon a tiny server instantly at the location you want.
    • No other platform allows you to horizontally scale so trivially. Not at this scale.
    • Fly.io is in my opinion the only other platform that has a nice UX doing similar kind of horizontal scaling with their Machines API offering. However, even in that case, managing the exact location of each instance, creating and tearing them down instantly, referencing them by a constant name/ID throughout the whole lifetime of your application (e.g. across blue/green deployments), and doing all of that transparently within my code without a lot of boilerplate, is not as seamless as just creating a Durable Object Stub.

  3. The Cloudflare Developer Platform.

    • I only used Workers and Durable Objects for the whole Tiddlyflare product.
    • There is so much you can do within the platform, and everything integrates with Workers through Bindings so nicely (and will get even more seamless).
    • Workers KV, R2, Workers AI, Cache API, Rate Limiters, upcoming Containers, and so much more.
    • Example: Adding a global cache to Tiddlyflare is simply env.KV.put(wikiId, src). This will now allow fast reads from any Worker processing requests for that wikiId without having to even reach the DO instance. 1 line!
env.WIKI.get(extractDOID(wikiUrl)).deleteWiki(wikiId);

That’s all it takes. 🚀

Mindset shift

Having said all that, I want to point out an adoption issue with a platform like Workers and Durable Objects. We are transparent grown up adults, after all.

Durable Objects have been public for several years now, but most developers have no clue what they are, and what they can do.

That’s because it’s a radically different combination of compute and storage infrastructure product compared to existing platforms.

I mentioned before that someone familiar with any Actor programming model (Elrang, Elixir, Akka) will feel right at home with Durable Objects.

It’s the Actor model with infinite scale built natively into the global Cloudflare network itself.

In the end, this boils down to thinking about the main entities of your application as separate “things” that communicate with each other.

Each entity has their own memory, their own local disk storage, and their own lifecycle. In our example above, we have two main entities, the tenant (user information), and the wikis (versions of HTML content).

At the application level we have 1 TENANT instance per user, and unlimited amount of WIKI instances per user.

You could model all the wikis to be managed by a single Durable Object instance, or further merging the two and only having one Durable Object instance for each user with all their information including wiki content. 👎🏼

The problem with that design is that all requests and all operations for a user, including all the operations for all of their wikis, would be handled by a single Durable Object instance. What if a single wiki gets DDoSed and then blocks all others? What if the machine hosting that single Durable Object instance goes down? What if…?

Durable Objects are a very, very powerful programming model. However, a single Durable Object instance is a tiny server of 128MB memory and 10GB disk space. There is only so much it can do for you and requests it can handle (up to 1 thousand per second).

I previously described a bunch of other example cases and how Durable Objects can influence their architecture.

In summary, the main hurdle of adopting the Workers platform is that you need to start designing your applications with this isolation in mind.

What if every core entity in your application had its own mini server?

That’s Durable Objects.

Conclusion

You made it. Awesome. Thank you. 🙏🏻

I hope you now know what Durable Objects are, and understand why they are powerful.

If you want to use them in your applications, and have more questions, please, please reach out. If you have feedback to improve the platform, please reach out.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Skybear.NET Scripts secret variables, HTTP triggers, and replacing AWS Lambda with Fly.io - Changelog 2024-10-15]]> https://www.lambrospetrou.com/articles/skybearnet-scripts-changelog-2024-10-15/ skybearnet-scripts-changelog-2024-10-15 Tue, 15 Oct 2024 00:00:00 GMT Table of contents

Skybear.NET is a managed platform automating your HTTP API testing using Hurl.dev plain text scripts. You can run your scripts on-demand through the API, or periodically based on a cron expression.

It’s been a few months since the last changelog article (see last update in July), but it doesn’t mean there wasn’t any work done. On the contrary, the features shipped since then are quite the bangers!

Let’s dive into them. 👇🏼

HTTP Hook Trigger

In a past update (see article) I introduced the script triggers, specifically the scheduled cron trigger. Scheduled triggers allow you to configure a script to run periodically every few minutes based on a cron expression.

However, in order to properly integrate with Continuous Integration (CI) systems running scripts periodically doesn’t work. We want to be able to run scripts on-demand, wait for the results, and then accordingly progress or abort the deployment.

Back in August (see changelog) I shipped a key feature, the HTTP hook trigger.

In addition to the Scheduled Cron trigger, you can now configure your scripts to be invokable through an HTTP POST request.

The request is blocking, and does not return until your script execution is complete. This allows you to run your scripts in your CI, and make sure any code changes you shipped are actually correct.

Skybear.NET http trigger settings
Skybear.NET script HTTP hook trigger settings.

You can see above the new script setting to enable your script’s HTTP hook trigger.

Please note that this URL should be treated as a secret, since anyone with that URL can send an HTTP POST request to it and trigger a run of your script.

Script runs triggered by the new HTTP hook trigger have a new indicator:

Skybear.NET http trigger run result
Skybear.NET script HTTP hook trigger run result.

Secret Hurl variables

Hurl scripts are very flexible, and they support variables in order to allow dynamic scripting. For example, the following script defines the variable BASE_HOST that needs to be provided at execution time.

GET https://{{ BASE_HOST }}/some-api
HTTP 200

With the release of the HTTP hook triggers, it was only natural that folks wanted a way to provide Hurl variables to their scripts since now they could run either through the UI, through their CI, or just manually through the HTTP trigger.

Skybear.NET shipped support for Hurl variables as part of the request to the HTTP hook trigger URL (see changelog), enabling fully dynamic scripts.

An example POST request to the HTTP hook trigger of a script:

curl --request POST \
  --header "Content-Type: application/json" \
  --data '{"hurlVariables":{"BASE_HOST": "skybear.net"}}' \
  https://api.skybear.net/v1/integrations/triggers/http/s_nkjfLc27FDb1dRz7rZ95zcc/strig_http_l9qRWlr16M3jm1LnbTzM7XtSNcGKShGtq:sync

A few days later, I fully integrated Hurl variables into the Skybear.NET platform and built a UI to manage these variables through the UI (see changelog).

You can define them in your Skybear.NET account secrets page, and use them across all your scripts.

Skybear.NET account secrets and variables
Skybear.NET account secrets and variables.

All Hurl variables are being treated as sensitive secrets. Never logged or stored in plaintext. 🔐 Script run reports are also encrypted.

I do envelope encryption. Each variable is encrypted with its own Data encryption key (DEK), that itself is encrypted with a different Key Encryption Key (KEK) for each account, and that KEK finally {en,de}crypted by AWS KMS!

I will write up a more detailed article on how envelope encryption is implemented.

With Hurl variables integrated natively, you can literally take your local scripts, put them in Skybear.NET, run them in the UI, configure them to run automatically every few minutes, or even get an HTTP URL that you can invoke in your CI and have them running in the Skybear platform remotely.

Full script compatibility with local Hurl CLI. 👌

Out of AWS Lambda into Fly.io

Back in September, a user reported that their script sending requests to an IPv6 origin was failing.

At the time, the script execution was done inside AWS Lambda. After a bit of research and testing, it turns out that AWS Lambda does not natively support outgoing IPv6 connections, therefore all scripts attempting to do that were failing.

I immediately created a “known issue” in our documentation (see issue entry) for this so that I can share it with other customers in case they encounter this.

I went through a few solutions, and decided that it was not worth the effort to fiddle with AWS networking shenanigans, having to move my AWS Lambda functions inside a VPC, creating and proxying all traffic through NAT gateways, etc.

After 2 days, all of the script executions were moved out of AWS Lambda into Fly.io (see changelog).

The transition was trivial, uneventful, and has been running smooth since then! 🥳

I have used Fly.io in the past, so I was familiar with it, and since the execution server was already an HTTP API written in Go it was trivial to migrate it from AWS Lambda into Fly.io with a tiny Docker container.

As a fun side-effect, I now use 3 regions on Fly.io instead of just one I previously had with AWS Lambda, therefore there is more resiliency in case any of their datacenters goes down.

And soon, I will enable users to select the location(s) where their scripts will be running. 🥳

Response bodies automatically persisted

Another huge upgrade in September was the upgrade of Hurl to version 5.0.1 (see changelog) that added the --report-json argument.

Before that upgrade, you had to use the output: <filename.extension> option in order to persist responses in your script run reports.

With the new Hurl option --report-json, the platform now automatically persists every* single response body received while executing your script, uploads it to durable object storage, and makes it available to you for download later (see changelog).

The output option continues to work, but you shouldn’t need to use it.

Debugging your scripts and APIs was never easier 🙃

* There is a limit on how much storage each script run can use at the moment, but I plan on extending this based on the active billing plan.

Docs

Finally, last month I released the initial documentation website for Skybear.NET.

Visit https://www.skybear.net/docs/ and let me know what you think.

It’s quite barebones at the moment, but I will be adding a lot of content over the next few weeks. Tutorials, existing feature descriptions, and plenty of examples for Hurl scripts.

Keep an eye on the Changelog page listing major and highlighted releases.

Conclusion and feedback

Check out Skybear.NET and let me know if you have specific feature requests.

If you have any questions, email me, or reach out at @lambrospetrou. 🙏🏼

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Durable Objects (DO) — Unlimited single-threaded servers spread across the world]]> https://www.lambrospetrou.com/articles/durable-objects-cloudflare/ durable-objects-cloudflare Wed, 14 Aug 2024 00:00:00 GMT Table of contents

In this article I will showcase Durable Objects (DO), probably the most underrated compute and storage offering by Cloudflare.

I am not going to focus on how you use Durable Objects in code at all, since the goal is to explain why Durable Objects should be used more, but you can find code examples in the developer docs.

TL;DR Durable Objects (DO) are built ontop of Cloudflare Workers (edge compute). Each DO instance has its own durable storage persisted across requests, in-memory state, executes in single-threaded fashion, and you decide its location. You can create millions of them!

If the above sentence is not clear, keep reading. You won’t regret it.

Not just for real-time collaboration

As of the time of writing, the Cloudflare developer documentation (see here) and the Durable Objects landing page (see here) describe DOs as follows:

Real-time, low-latency API coordination and consistent storage

Durable Objects provides a powerful API for coordinating multiple clients and users — helping you build collaborative applications while maintaining strong consistency of state. […]

Durable Objects provide a powerful API for coordinating multiple clients or users, each with private, transactional and strongly consistent storage attached.

In my opinion, even though the above is accurate, it focuses too much on the real-time collaboration use-case and throws off customers that are not actually working on real-time collaboration software.

That is a lot of people, including myself when I initially read about Durable Objects (DO).

I will showcase why Durable Objects are a great fit for a lot more use-cases than real-time collaboration.

Workers intro

Durable Objects are super-powered Cloudflare Workers (see docs [1] [2] [3]).

Workers are Cloudflare’s serverless compute platform that:

  • Doesn’t have cold starts.
  • Runs in more than 300 locations across the world.
  • Supports Javascript and any WASM-compiled language.
  • Has direct access to all of Cloudflare’s infrastructure (see all Runtime APIs).

You can read more on how Workers are implemented on-top of V8 isolates in the excellent “How Workers works” page.

The things we care about for this article is that Workers:

  • Are limited at 128MB of memory, therefore hard to do in-memory caching for anything significant.
  • Concurrently handle many requests on a single Worker instance (V8 isolate).
  • Are stateless across requests, and each request can potentially be routed to a different machine in Cloudflare’s network.
  • They can be destroyed and recreated whenever, therefore user code should have no assumptions about affinity (i.e. where it runs) or longevity (i.e. how long each worker instance runs).

Durable Objects intro

Now that we know what Workers are, we can better understand Durable Objects.

In my mind, Durable Objects (DO) are Workers with durable storage, but there are some important differences from Workers.

Firstly, let’s go through some key properties of DOs:

  • Built on-top of Workers, so they support exactly the same code (Javascript+WASM), with the same memory limits.
  • Each Durable Object instance stays alive for several seconds before hibernating, hence allowing in-memory caching to boost performance (see docs).
  • Each Durable Object instance has its own local durable storage that only that specific instance can access (read/write) (see docs).
  • Each Durable Object instance has an identifier, either randomly-generated or user-generated, allowing you to “select” which Durable Object instance should handle a specific request or action by providing this identifier (see docs).
  • They are not available in every location like Workers, but are still spread around the world (see the where.durableobjects.live project for live locations), and most importantly you can influence where each instance should be located (see Location hints docs).
  • They provide an Alarms API that allows you to schedule an execution of your Durable Object instance any time in the future with millisecond-granularity (see docs).
  • They are effectively single-threaded; when a request execution causes any side-effects (e.g. durable storage reads/writes) other requests to that specific DO instance are blocked until the request completes. Read more details on how this is implemented in the blog post “Durable Objects: Easy, Fast, Correct — Choose three”.
  • Each Durable Object “type” (or “binding” in Cloudflare terms) maps 1-to-1 with a Javascript class implementing the business logic. You can create unlimited instances of each Durable Object type.

There are more things to know, but I listed the key features I care about.

You probably already spotted some important differences between Workers and Durable Objects:

  • Fully stateless (Worker) vs In-memory state (DO)
  • No instance affinity (Worker) vs Affinity by an identifier (DO)
  • Always near the request (Worker) vs Ability to influence location (DO)
  • No storage (Worker) vs Durable storage (DO)
  • Concurrent execution (Worker) vs Single-threaded execution (DO)

Durable Objects and the Actor model

Another way of describing and thinking about Durable Objects is through the lens of the Actor programming model (🧍🏼‍♂️↔🧍🏻).

There are several popular examples of the Actor model supported at the programming language level through runtimes or library frameworks:

  • Erlang OTP used by Erlang and Elixir, probably among the oldest and most feature-rich Actor runtimes.
  • Akka for Java, Scala, and later .NET.
  • Microsoft Orleans for .NET.
  • more…

Each Durable Object instance can be seen as an Actor instance, receiving messages (incoming HTTP/RPC requests), executing some logic in its own single-threaded context using local durable storage or in-memory state, and finally sending messages to the outside world (outgoing HTTP/RPC requests or responses, even to another Durable Object instance).

The Actor model simplifies a lot of problems in distributed systems because it abstracts away the communication between actors using RPC calls that could be implemented on-top of any transport protocol, and it avoids all of the concurrency pitfalls you get when doing concurrency through shared memory (e.g. race conditions when multiple processes/threads access the same data in-memory).

The most astonishing feature of Durable Objects, and the super differentiator from the above Actor frameworks, is that with Durable Objects you get all of the above, built-in, while having a huge distributed network to spread your actors across. 🤯

When I was at WhatsApp a few years ago, I was really amazed by how the Erlang runtime simplified and solved many distributed systems problems for us. Each chat (e.g. between 2 people) was handled by a single Erlang process (a single actor), and all messages of that chat were routed to that specific process using the Erlang runtime primitives for routing. This meant that all the logic for each chat was running in a single-threaded process with normal sequential code. Easy to reason about. We also had tens of thousands of servers across the US and Europe connected in a mesh so that the Erlang runtime can route messages across them.

With Durable Objects you get all of that 💪🏼 But with orders of magnitude more power and much less effort than if you had to build all this infrastructure yourself!

Your system now comprises of Cloudflare’s whole network. Even though Durable Objects are not in every location of the CDN like Workers, they are in more than 25 cities across the world as of today (see where.durableobjects.live) and they will keep expanding over time.

As mentioned above, you can create unlimited instances of each Durable Object type (or binding), therefore if you design your Actor model well, you can scale across the whole of Cloudflare’s edge network.

Durable Objects use-cases

Now that we know all about Durable Objects, let’s see how they can be used to simplify your architecture.

As a guideline, if you have any use-case where there is a clear boundary between some or all of your resources, you can use Durable Objects to simplify your logic when processing and storing each of those resources.

It can be per user, per chat channel, per factory warehouse, per object storage bucket, per workflow, per tenant in a multi-tenant SaaS, etc.

If all your database queries have a WHERE resourceID = 'abc' in order to restrict your queries to the correct subset of your data with that resource identifier, then Durable Objects could potentially simplify your life. It always depends on the concrete use-case obviously, but this is the starting point to see if they can benefit you.

Real-time collaboration

If it’s not yet clear, the reason DOs are awesome for real-time collaboration, is that you need “total order of events” such that all the “actors” collaborating end up in the same final state.

Chat systems

Each chat channel (e.g. Lambros talking with Ben) is one DO instance, therefore all messages go through the same instance and processed sequentially.

The volume for a single channel is relatively very low, but the volume of all messages across channels is usually humongous. This is perfect for Durable Objects, since you can have unlimited DO instances spread across the world (closer to each chat channel’s members).

Imagine that the Durable object instance responsible for the chat between Lambros and Ben is in London/UK, but the instance for the chat between Paul and Jasmine is in Portland/US.

Online document live-editing

Each document is one DO instance, therefore all edits of the document are processed sequentially by the same DO instance and stored in the instance’s durable storage.

With unlimited Durable Object instances, you essentially support unlimited number of documents.

Multi-tenant SaaS

Imagine you are building a restaurant booking system, or any other booking system.

You need to process bookings in-order to avoid double-booking tables, and usually you do this with database transactions.

If your volume is low, then a single-server database is fine, and you should go with that. But what if you want to grow and expand your service to thousands of restaurants, or even across the world and you want to be local to each of those regions (latency, jurisdiction regulations, etc).

Durable Objects can help you. Each Durable Object instance can handle all bookings for a single restaurant, and you can have unlimited number of DO instances hence support for unlimited number of restaurants.

CI/CD pipelines

Imagine you are building a Continuous Integration (CI) and Continuous Deployment (CD) service.

You need total ordering not only across pipeline executions, but also durable state persistence for each individual execution in case they can be paused and resumed later or just to keep track of progress and reference artifacts.

If you want to have parallel independent executions of a pipeline, each pipeline execution can be represented by a Durable Object instance that keeps track of each execution, and one instance that just keeps track of all the execution IDs and maybe coordinate them (e.g. restrict how many parallel executions you can have).

Alternatively, if you want to restrict only one execution at a time for each pipeline stage, but want to allow multiple executions to exist in different stages of the pipeline (similar to Amazon’s Pipelines) you can model your pipeline as a Durable Object instance that keeps track of all the pipeline executions of the pipeline.

The actual modelling depends on your requirements, but you can see where this is going.

So many more

I cannot enumerate every single use-case out there…

The gist is that if you can think of a nice boundary across your resources, there is a high chance for Durable Objects to be very useful to you.

In Cloudflare we use Durable Objects for a lot of the internal and external products and more are adopting them every day.

Limitations

You can find all the limits in the Durable Objects documentation, but the main things to keep in mind are:

  • Throughput of each Durable Object instance (see docs).
  • Total storage per account (can be raised by contacting support).
  • Size limits for the storage’s keys and values.

Most Worker limits apply too (see docs).

Pricing Durable Objects vs Workers

Just pointing out that pricing for Durable Objects is different than Workers.

Workers are charged purely on CPU time (see docs), whereas Durable Objects are charged on number of requests and duration (see docs).

As mentioned previously, Durable Objects can stay alive and have in-memory state, hence why the duration dimension is taken into account for pricing.

Almost every serverless compute provider is also charging by duration so it shouldn’t be surprising, but something to keep in mind when comparing with Workers, famous for CPU-only pricing.

Conclusion

Durable Objects (DO) are a super powerful tool. When they fit a use-case, they can simplify so many things while keeping the developer experience great and operational overhead low.

If you can think of a nice boundary across your resources or users, there is a high chance that Durable Objects will be very useful to you.

Each Durable Object instance is identified by a user-provided key, has its own throughput limits, its own durable storage, its own in-memory state, executes in single-threaded fashion, and you can influence its location if necessary.

Oh, and BTW, there is nothing blocking you from using Durable Objects for the resources that fit that model, and another storage product (e.g. D1 SQL database) for your highly relational resources.

You can find a few select Durable Object examples on the developer docs as well.

If you have feedback about the article, or Durable Objects, feel free to reach out 😉

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Brag document and folder — feels good only]]> https://www.lambrospetrou.com/articles/brag-doc-folder/ brag-doc-folder Sat, 27 Jul 2024 00:00:00 GMT Table of contents

In this article I explain what a “Brag document” is, how it relates to my own “Worklog document”, and how I use them both.

Many definitions

A Brag document has many names, depending on who you ask, and it can also contain a variety of content, again depending who you ask.

Julia Evans was among the first that popularized the term “Brag document”. Her post is really good, go read it too (after this one though 😅).

Others also wrote their own versions, mostly similar to Julia’s post.

Finally, several folks often discuss about this type of documents on social media, e.g. this Reddit post titled “Keep a Brag Document”.

A Brag document is many things, but the common definition is:

  • Listing what you do over time at work, tasks, problems solved, etc.
  • Making it easier to find your achievements for promotion docs, interview dicussions, etc.
  • And can use it as a source of happiness and feeling of achievement.

Is this… the Worklog?

Several years ago, I wrote an article describing the “Worklog document”, my own kind of document I was keeping throughout my career.

If you read my article, and compare it with the articles referenced above, you will see a huge overlap. I didn’t know the term “Brag document” back then, but they look awfully similar.

This is why I use the term “Brag document” a bit differently than the above.

Brag document and folder

I personally use a Brag document just for the good stuff, the achievements, and the praises.

It’s not a list of things I do over time, it’s not a list of projects I delivered, and it’s not the number of oncall incidents I managed to resolve. It is complementary to my Worklog.

It has screenshots of chat messages from colleagues thanking me for something.

It has quotes from others praising my work, in chat, in announcements, in wikis, or given as part of performance review feedback.

It has awards I have won, or significant milestones in my career.

It is a document, and literally a folder of screenshots.

You can see some of these screenshots in my 1:1 Tech Interviews page, with colleagues praising my article about software engineering interviews and asking if they could use it themselves.

A quote from my last performance review feedback can be found also in my article “Ownership - High agency - Manager of One” (go read it and find out which one 😎).

Conclusion

This Brag folder and document is literally the place where you can go and just feel nothing but good about yourself, your work, and your achievements.

Everyone should keep a Brag document, regardless of level or industry, from Juniors to Distinguished Engineers, to doctors, to teachers.

All people forget. Colleagues and managers, and even you, will forget what you did in the past. This is your way of persisting the great things you do forever.

Trust me, your future self will thank your current self for doing this.

Do good work, and keep updating your Brag document and folder. 🥳

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Skybear.NET Scripts landing page and Business plan - Changelog 2024-07-21]]> https://www.lambrospetrou.com/articles/skybearnet-scripts-changelog-2024-07-21/ skybearnet-scripts-changelog-2024-07-21 Sun, 21 Jul 2024 00:00:00 GMT Table of contents

Skybear.NET is a managed platform to automate your HTTP API synthetics testing using Hurl.dev plain text scripts.

Use it for testing your HTTP APIs on-demand or periodically, or use it as a complex orchestrator for a sequence of HTTP requests that need to be executed in order using data from previous ones at specific times of the day.

The past two months was a lot of work, but also a few weeks of holidays to recharge. Let’s dive into the changes that happened.

Run types

In the last update (see post) I introduced the script triggers, specifically the scheduled cron trigger. This type of trigger allows you to configure a script to run periodically every few minutes.

I also introduced the historical script runs listing page where you can see all the invocations of your script, either manually triggerred through the UI, or due to the scheduled cron trigger.

For this changelog, I did a small user-experience improvement to denote the type of each run.

Skybear.NET script run invocation type in list runs page
Script run invocation type in list runs page.

A small indicator now exists in the scripts list page that is awesome in quickly checking out which of your scripts are scheduled to run periodically, and how often.

Skybear.NET script cron trigger configuration in scripts list page
Script cron trigger configuration in scripts list page.

Landing page

I finally launched the product’s landing page. 🥳 Check it out at https://www.skybear.net/.

I spent some time working on the landing page’s copy, trying to emphasize what Skybear.NET is about, what you can use it for, and why it’s not simply yet another uptime checker.

This is only the beginning though, and I have a lot of changes coming up soon that will enrich the landing page even more. Take a look, and let me know if I can improve something!

Skybear.NET product landing page
Skybear.NET product landing page above-the-fold.

While working on the landing page, I used several resources as guidance, especially a few products I love using or following (e.g. Tailscale, Tailwind CSS, Postmark, and anything from Basecamp). You can spot some of their influence in my landing pages 😅

The most helpful resource though, which I recommend to anyone working on landing pages to read, is this guide by Julian Shapiro: Resource: Landing Pages

Go ahead and read it, now! Thank me later 👌

Business plan

After a few weeks of holidays, and several days of work, I released the Business pricing plan for Skybear.NET.

I spent more time than I wanted reading about pricing, checking out how competitors price their plans, how multiple other relevant and irrelevant products do their pricing, and went through multiple versions for my own.

A few months ago, I actually wrote my thoughts on SaaS pricing, so I had that as my guarding rails as well.

I decided for now to go with a usage-based pricing using the total number of step requests across all your scripts, combined with a volume-based bulk discount approach. Instead of going fully usage-based where you pay only what you use, I decided to use 7 price points that I consider sufficient for companies and teams at different growth levels, and apply them a discount as you go higher.

Right now the prices available start from 15 USD/month and go up to 1100 USD/month, with an annual plan offering a whole of 2 months for free.

There is also a completely FREE tier, albeit much restricted, that someone can use to try out the platform before switching to the recurring monthly or annual plans.

Skybear.NET product pricing plans
Skybear.NET product pricing plans as of 2024-07-21.

Are these the final and only plans forever? … Probably not… I would lie to myself, and you, if I said pricing won’t change (higher or lower) once I start getting customer feedback, or experimenting a bit more with it, adding more features, etc.

Please try out the platform, and let me know if you have suggestions on pricing.

Continuous Integration use-case

The core platform features, including the landing page, pricing integration and respecting the plan limits, are now implemented.

In the next couple of months I am going to focus on expanding the platform’s feature set around Continuous Integration (CI).

Many teams and companies use Synthetic API Tests to verify their code changes before shipping them in production. There is a big userbase using Hurl.dev scripts or similar tools like Bruno, and many of these teams run their tests in CI jobs in addition to their local tests during development.

I would like to dive into this market base and provide features that would entice these teams to use their existing scripts and test files with the Skybear.NET platform in order to get more visibility into their tests, and other value-adding features.

There is already a big list of ideas I am brewing and will start working on over the next couple of months, but if you are using any of these tools and have specific needs for what you need out of a platform to run your tests, please reach out.

Conclusion and feedback

Skybear.NET can already be used for real-world use-cases. 🚀

I use Skybear.NET to test Skybear.NET! Check it out, try to run your own scripts on-demand or periodically, and let me know if you have specific feature requests.

If you have any questions, email me, or reach out at @lambrospetrou. 🙏🏼

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Skybear.NET Scripts response bodies and cron triggers - Changelog 2024-05-28]]> https://www.lambrospetrou.com/articles/skybearnet-scripts-changelog-2024-05-28/ skybearnet-scripts-changelog-2024-05-28 Tue, 28 May 2024 00:00:00 GMT Table of contents

Skybear.net Scripts is a managed platform to automate your HTTP website and API tests using Hurl.dev scripts. I like to call them HTTP workflows.

Use it for testing your HTTP APIs periodically, use it as a website uptime checker, or use it as a complex orchestrator for a sequence of HTTP requests that need to be executed in order using data from previous ones at specific times of the day.

Let’s dive into the changes of the past few months.

Full CRUD

In the last update (see post) I introduced the management of Skybear.net scripts: creating, updating, and listing.

A few days later, the delete functionality was completed and rolled out as well (see tweet).

Skybear.net Scripts Delete script action
Delete script action.

Hurl 4.3.0

I updated the Hurl version used for running the scripts from 4.1.0 to 4.2.0 (see tweet - see changelog), and a few weeks later again to 4.3.0 (see tweet - see changelog).

Staying up-to-date with hurl means more features and bug fixes to all Skybear.net users.

Response outputs

Starting with Hurl 4.3.0, there is a new output: <filename> option for each “entry” in the Hurl script, denoting the filename into which to save the full response body of the request.

You can now have full end-to-end introspection of the request and its response, including status code, headers, and body.

As an example, the following snippet will query the httpbin.org/headers endpoint, and save its response into the headers.json file.

GET https://httpbin.org/headers
[Options]
output:headers.json

The screenshot below shows a more complicated workflow involving 4 HTTP requests, each with its own resource output file. Clicking any of the four filenames (see Output resources section) opens the corresponding file as-returned in the original response.

Skybear.net script with multiple requests saving their responses using the `output` option
Multiple requests saving their responses using the `output` option.

Historical script runs

Skybear.net scripts are now powerful enough to cover complex scenarios and use-cases. Many times, it’s useful to access past runs of a script and examine its results, either for comparing with a more recent run, or just as a reference during incident investigations.

You are now able to access the full script run reports from the past 30 days. This retention period will be configurable with upcoming paid plans (get in touch if you have special needs).

Skybear.net View past script run results actions
View past script run results actions.

Script triggers - Scheduled Cron

All the above improvements led to the new script triggers feature. Triggers are numerous mechanisms leading to an execution (aka “run”) of your scripts.

Last week I released the first among many, the scheduled cron trigger.

You can now specify how often to run your scripts, with 1-minute granularity.

For now, you cannot configure a script to run more often than every 10-minutes, but upcoming paid plans will allow you to run scripts as often as every few seconds! 🤯

This feature was built ontop of Cloudflare Durable Objects allowing for fine-grained second level granularity which provides plenty of flexibility for future extensions!

Cloudflare Durable Object logs for Skybear.net scripts cron triggers
Cloudflare Durable Object logs for Skybear.net scripts cron triggers.

Soon, there will be a built-in notification mechanism as well to notify you when the script fails to successfully complete execution.

Check out the video below for a showcase of configuring a scheduled cron trigger.

Video showcasing the Skybear.net Scheduled Cron Triggers feature.

Conclusion and feedback

Skybear.net Scripts can already be used for real-world use-cases. 🚀

I use Skybear.net to test Skybear.net!

If you have any questions, email me, or reach out at @lambrospetrou. 🙏🏼

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Ownership - High agency - Manager of One]]> https://www.lambrospetrou.com/articles/ownership/ ownership Sun, 07 Apr 2024 00:00:00 GMT Table of contents

Definitions

What is holistic ownership of a project?

Holistic ownership. Person ABC, in the projects we have worked together, has excelled at holistic owning the problem (beyond specific projects or tasks). This is particular important when dealing with large scope problems where stepping back and looking at the overall picture to propose next steps is very important.

How important is ownership in leadership?

On any team, in any organization, all responsibility for success and failure rests with the leader. The leader must own everything in his or her world. There is no one else to blame. The leader must acknowledge mistakes and admit failures, take ownership of them, and develop a plan to win.

Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job”.

The 37signals folks advocated for hiring managers of one back in 2008.

A manager of one is someone who comes up with their own goals and executes them. They don’t need heavy direction. They don’t need daily check-ins. They do what a manager would do — set the tone, assign items, determine what needs to get done, etc. — but they do it by themselves and for themselves.

Having ownership and being a manager of one is very closely related to “high agency”.

High Agency is a sense that the story given to you by other people about what you can/cannot do is just that - a story. And that you have control over the story. High Agency person looks to bend reality to their will. They either find a way, or they make a way. Low agency person acepts the story that is given to them.

When you’re told that something is impossible, is that the end of the conversation, or does that start a second dialogue in your mind, how to get around whoever it is that’s just told you that you can’t do something?

High Agency is about finding a way to get what you want, without waiting for conditions to be perfect or otherwise blaming the circumstances.

All the above, except one (😉), are definitions given by people for ownership, high agency, and being a manager of one. There are many other terms referring to the same thing.

Some companies like Amazon put this in their leadership principles and company values. Others, put this as an explicit dimension in their performance evaluation criteria (most of the tech companies at least).

This particular skill, owning some problem end-to-end, is a key ingredient for success in business, family, and personal development alike.

Applicability and importance

Throughout my career, I have been putting a lot of effort in developing my ownership skills.

I always try to understand a problem holistically, think about problems and edge cases, design a solution, and ultimately make sure the solution rolls out successfully.

Does this apply to tech jobs though, or only software projects? Absolutely not!

Actually, one of the previous quotes is from Jocko Willink’s book “Extreme Ownership: How U.S. Navy Seals Lead and Win”. He explains how being a leader with ownership within the US Navy SEAL forces is crucial. And he was literally in “life or death” situations.

Check his talk below for a short glimpse of his perspective on extreme ownership.

Being a person with the quality of owning whatever you do, having high agency to being able to navigate ambiguities and the unknowns, is so crucial that everyone can benefit from.

Are you working in tech? Are you a surgeon? Are you an investor? Are you a trader? Are you just a parent struggling with the kids?

You need to be able to identity problems and understand them end-to-end. You need to be able to find solutions, and ensure they are implemented. You need to be able to acknowledge mistakes and take responsibility in order to improve things going forward.

Everyone has colleagues that are just waiting to be told what to do all the time, without taking any initiative or having any bias for action.

Many middle managers usually hide within the bureacracy of big organizations, acting as dumb passthough proxies from reports to superiors and vice-versa. Most times, even doing a horrible job at that, when they filter out the wrong things. Never owning any of the problems hurting the team and never taking actions to improve things.

There are also folks that can never acknowledge they are wrong. Everyone makes mistakes. That’s how we learn as a species. Having ownership, means you own the mistakes too.

It doesn’t even have to be that you individually made the mistake, but that someone on your team did. If you are leading a project or a team, and turns out that better guidance or better communication could avoid the mistake, then you share the mistake.

A leader should take responsibility, and find ways to improve a bad situation going forward.

Conclusion

Fun fact: As I hinted earlier, all the quotes at the top are definitions I found from online sources, except one. One of the quotes is verbatim feedback I copied from one of my performance evaluation reviews. Can you figure out which one? 😉 Shamelessly self-bragging.

Be an owner. Have high agency. Be a Manager of One.

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[SaaS Pricing - What I want vs What I offer]]> https://www.lambrospetrou.com/articles/pricing-want-vs-offer/ pricing-want-vs-offer Sun, 18 Feb 2024 00:00:00 GMT Table of contents

Pricing for a SaaS, or online digital products in general, is a hot topic for debate. All the time.

In this article I explain the pricing model I like as a customer of a SaaS product, and then compare that to the model I would (will) offer in my own SaaS product. Are there any differences? Why?

Context

Last year I spent several months contemplating starting a PaaS business, a managed infrastructure platform. I decided not to in the end, but I decided to write about several things I noticed in an article.

One of the aspects I researched was the pricing models different PaaS businesses used (see Pricing section).

Pricing is a huge thing for any company. Pricing defines, and filters, the customers a company will attract. Do you want the Enterprises, charge thousands. Do you want the indie developers, give stuff for free. The pricing model is what will make a company profitable, or kill it.

The above is very true, and drives the pricing model of a company. 👆🏻

I then elaborated on what I prefer as a user too. 👇🏻

[…] for me, as a customer, I like it more when pricing starts from zero ($0) when I don’t consume any resources, and increases as I use the platform more.

Who would have guessed… Someone that wants to pay as little as possible.

Not really though! I want to pay as close to nothing as possible, when my use of a product is zero. When I use a product, I want to pay proportionally to that usage.

And then wrote the following two-fold model as my ideal pricing model:

I thought about pricing a lot. The model I would choose if I ever built it would be two-fold:

  • Provide usage-based pricing that starts from $0, but with a higher price-per-unit.
  • Provide per-seat or volume-discounted pricing for bigger companies.

With the above two-fold model, anyone can start using the platform to make sure it works for them, pay for their usage along the way, and once they settle to use it they could switch to the second pricing plan. Some prefer predictability, some prefer usage-based, so why not both.

I read a lot of pricing material over the past few years (see #References section below for some of my favourites). There are people arguing in favor of anything and everything.

What I want vs What I offer

The above two-fold approach is my personal favourite model, as a consumer, and (I think) as a business.

I have seen folks that complain about fixed high prices for a product, or talk negatively about too many recurring subscriptions, for them as an individual consumer. But then, their own business only offers those too. If your own business is not offering to others what you like to get, why do you expect other businesses to do so.

I am just starting a paid product business (see Skybear.net Scripts), and its pricing will be two-fold as above. I am putting my skin in the game, or if you prefer, I will be walking my talking.

I want to provide a usage-based/credit-based pricing plan for those that don’t want recurring monthly charges. I also want to offer bundled/tiered plans for those that prefer predictable per-month/per-year charges. Doing this without confusing users and leading them to comparison math is the interesting bit. To be explored.

Hopefully this article will not age badly 😁

Usage-based pricing

There are several pricing models that I put into this category, including (and not only):

  • prepaid credits (prepay for X credits, and spend them whenever)
  • usage metered / pay as you go (pay at the end of the month for how much you used)
  • percentage commission (pay a % of some revenue to the service provider)
  • more…

Why do I like usage-based pricing as a customer?

  • Pay as little as possible when I don’t use the product.
  • If my use is spiky, I don’t want to pay the peak all the time, or getting throttled when the spike happens because I am paying the lower plan.
  • I can use the product in the same way, for a toy project, as well as a huge for-profit project.

Drawbacks of usage-based pricing?

  • Harder to meter and track, for the provider. Need to find the right dimensions to bill for, without pushing customers to weird usage of the product when cost-cutting.
  • Harder to predict, for the customers.

Examples of usage-based pricing models:

  • Amazon Web Services (AWS)
    • Although AWS takes it to the extreme often, and makes it impossible to actually calculate your bill. They don’t have to. They choose to.
    • AWS CodePipeline charges per active pipeline (v1), or per execution minute(v2).
    • AWS Lambda charges per execution GB-second.
    • Amazon S3 charges per GB-stored, and per request.
  • Upstash
    • Entirely usage-based on a dimension that makes sense per product (e.g. number of commands for Upstash Redis)
  • Stripe
    • Commission on every $ they process for you.

Per-seat - Recurring pricing

Pricing models that I put into this category, including (and not only):

  • Per-seat price per month (usually $$ per employee/user)
  • Tiered plans with fixed charges per month
  • Bundled volume-based plans with fixed charges per month
  • Recurring subscription-based with fixed charge per month
  • more…

The common aspect in these models is that you choose among a few predefined plans, and then you pay that much per month or per year regardless of how much you used the product. These plans often have an additional dimension for overages in order to cover unexpected usage that exceeds the allotment of the chosen plan.

Why do companies like to pay or offer predictable/fixed pricing?

  • Easier to reason and plan for, considering some providers (e.g. AWS) make it very complex to understand your bills.
  • Per-seat plans are often more manageable to manage in terms of licensing, control it over time depending on active employees, etc.
  • MRR is trendy.

Drawbacks of recurring pricing?

  • See benefits of usage-based pricing above.

Examples of recurring pricing:

Conclusion

Both pricing approaches have pros and cons.

I like the flexibility of usage-based pricing, and at the same time I understand that the predictability and easier reasoning of fixed recurring pricing is preferred by larger businesses.

The fixed bundled per-month plans with overages is a hybrid actually, but usually its high floor pricing doesn’t bring the benefits of usage-based pricing when your usage is lower than those.

In my opinion, we should do both.

I like for example how Upstash does it (as of now at least), with usage-based pricing starting at $0, and then becoming more cost-effective to switch to bundled/fixed pricing per month once your usage exceeds a certain threshold.

References

This section lists some of my favourite material for product pricing.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Skybear.NET Scripts private user accounts - Changelog 2024-02-18]]> https://www.lambrospetrou.com/articles/skybearnet-scripts-changelog-2024-02-18/ skybearnet-scripts-changelog-2024-02-18 Sun, 18 Feb 2024 00:00:00 GMT Table of contents

Skybear.net Scripts is a managed platform to execute Hurl.dev scripts. I like to describe Hurl scripts as simple workflows that orchestrate a sequence of HTTP requests, while doing response assertions and transformations.

I blogged about the silent first release of an earlier version of the tool two months ago (see article), coined as Hurl Webscripts.

Rebranding to skybear.net

First update since that article, is that now the platform is hosted under the Skybear.net brand.

I own several top-level skybear domains, and this is perfect use-case for them 😅

The previously released https://webscripts.lambrospetrou.com now redirects straight to the Skybear.net Open Editor, so no existing shared links broke during this migration.

Skybear.net Scripts cover image
Skybear.net Scripts

User accounts

This was a beefy release and took several weeks to rollout publicly. It was in production behind feature flags with me the sole user though for a while now.

The main reason is that it introduces user accounts, and everything from now on revolves around these accounts in terms of ownership, storage, and access control.

Previously you could only work with the Open Editor and sharing your scripts with others putting the whole script source code into the URL as params. This was fine as an initial release, but in the end it’s not easy to iterate on a script, make improvements, and save those changes if you need to always get a new URL to store in your own records.

Now, with private scripts in your account, you can create as many scripts as you want, edit them and save them as many times as necessary to get them right, and they are persisted on the platform so you don’t have to worry about keeping track of your scripts.

It took a bit of time to fully roll this out cause I wanted to take into account future features too and implement it properly. Few things coming over the next couple of months include SSO authentication, teams and organizations, inviting users to read or author your scripts with you, and lots more.

It’s finally out though! 🎉 New features will now ship in a more regular cadence.

Skybear.net Scripts sign in page
Skybear.net Scripts sign in page.

Script management

The script management story is very simplistic at the moment, but complete, in the spirit of SLC (Simple, Lovable, Complete).

You can create scripts, update their source code, execute them, see the execution results, and finally list all your scripts. Deleting scripts is the obvious miss in that list, but don’t worry. It’s almost done, just needing a few final touches and it’s out next week.

I have many other plans for script management, but they will be added incrementally. I don’t want to release features that make the product feel incomplete when directly related features are missing, so I am bundling them in batches.

As the SLC article above explains, it’s better to develop a bicycle in v1, even if your v5 will be a car. You can already get value out of Skybear.net scripts without feeling out-of-place.

Skybear.net Scripts list page
Skybear.net Scripts listing page.

Coming very soon

User accounts and script management, were the two main features prepping the ground for everything else to follow.

I am so glad this release is finally out, and I can now gradually add features, much faster!

Some of the short-term features that are either already in-progress or will be soon:

  • Upgrade the Hurl execution engine to v4.2.0 from v4.1.0.
  • Support for names and descriptions per script for easier identification, in addition to the existing generated IDs.
  • Include all the HTTP responses of the script execution in the results, giving you complete visibility.
  • Add many more examples to make it easy to author Skybear.net scripts. (There are some fancy features I am brewing in this topic long-term…🤐)

Too many nice things coming up! ✨

Conclusion and feedback

Skybear.net Scripts is now ready for more complex Hurl workflows.

I already experimented with some workflows doing more than 300 HTTP requests (😮), asserting the responses, and making sure everything works correctly.

I want to see your own scripts, your own workflows, and your own use-cases. I am eager to hear your feedback and feature requests.

If you have any questions, email me, or reach out at @lambrospetrou.

Skybear.net Scripts release LinkedIn post
Skybear.net Scripts release LinkedIn post on 2024-02-18 (see post).
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Hurl Webscripts v0.0.1]]> https://www.lambrospetrou.com/articles/hurl-webscripts/ hurl-webscripts Thu, 28 Dec 2023 00:00:00 GMT Hurl Webscripts is a managed platform to execute Hurl.dev scripts. Hurl scripts do a lot, but to keep it short, they send HTTP/cURL requests, do assertions on the responses, and pipeline consecutive requests using previous responses.

I don’t want this to be a generic compute platform (e.g. AWS Lambda). I do want it though to grow into more use-cases, and if all goes well and according to plans, there are lots of exciting directions.

See a small showcase of the tool as of today. This is mostly for my historical records 😅

v0.0.1

I just released the very first version of the platform a few days ago (see tweet).

Initial version v0.0.1
Tweet of the initial version v0.0.1 release.

It only allowed you to type a Hurl script, run it, and see the generated reports as-is from the Hurl CLI. Albeit with some modifications and cleanup to make them to render well on my website.

A few improvements quickly followed up.

Shareable links (see tweet) allowing you to share the authored script with anyone so that you can share your masterpieces with more folks, on social media, or in your team’s Slack channels.

A new network requests timeline visual (see tweet) explaining the time each network operation took for every HTTP request of the script. You can clearly see how much time was spent in DNS Lookup, TCP Handshake, SSL handshake, waiting for server response, and transferring data.

Network requests timeline visualization
Network requests timeline visualization.

All code editors, at some point introduce some kind of split screen mode with multiple panes to allow for more things to be shown at once.

I implemented such side-by-side mode (see tweet), with the Hurl script code being on the left side, and the output on the right side. Size of each pane is configurable, and my underlying implementation supports arbitrary number of panes on each axis, so I will be doing a lot more enhancements in the layout department over time.

Side-by-side mode in mobile landscape
Side-by-side mode in mobile landscape.

Going forward

Now that the first initial version is up and running, I am able to ship smaller incremental improvements continuously. I have lots of ideas about features and different major directions this platform could go.

I will try a few things and see what will stick with the market.

There are some features I will be implementing either way:

  • Support for more Hurl options (see docs).
  • AI assistant and plenty of examples to help you write Hurl scripts.
  • Scheduled execution of scripts (e.g. run this every Tuesday at 09:00 and notify “this” email).
  • Unique invoking endpoint that will execute the script by just calling that endpoint.
  • Managing a collection of scripts and running them all together.
  • Source control versioned scripts (e.g. Github integration).
  • and many more…

I am not sure yet which major direction I will pick, but some areas I want to explore include:

  • Webhook integrations
  • Website and API scraping
  • API end-to-end testing
  • Adhoc HTTP testing
  • more to discover…

I am having lots of fun with this platform, and the more I build, the more fun it gets. Each feature I ship enables more use-cases, and makes the user-experience better, and this gives me joy.

Looking forward to productizing this soon! 💪😉

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[How to find listening and open ports in Linux]]> https://www.lambrospetrou.com/articles/network-ports-linux/ network-ports-linux Tue, 05 Dec 2023 00:00:00 GMT In many cases I want to check information about the network ports on my system, either to see what is using a specific port, or to see which ports are listening for requests.

Option 1 - lsof

lsof -nP +c 15 | grep LISTEN

The command lsof stands for “list open files”. It provides information about files and processes that are currently open on the system.

The following options to lsof make the command faster by avoiding DNS and service lookups.

  • -n prevents the conversion of network numbers to hostnames.
  • -P prevents the conversion of port numbers to service names.

The option +c 15 specifies the number of seconds to wait before giving up on network-related operations.

The | grep LISTEN pipe command filters the output of lsof for lines containing the word LISTEN, which indicates processes that are listening for incoming network connections.

Option 2 - netstat

netstat -tuln

The command netstat stands for “network statistics”. It provides information about network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

The command netstat -tuln is used to display information about active network connections and listening sockets.

The options -tuln to netstat mean:

  • -t specifies that only TCP connections should be displayed. It filters the output to show only information related to TCP protocols.
  • -u specifies that only UDP connections should be displayed. Similar to the -t option, it filters the output to show only information related to UDP protocols.
  • -l stands for “listening” and instructs netstat to display only listening sockets, which are endpoints for incoming connections.
  • -n prevents the conversion of numeric addresses to symbolic hostnames. It speeds up the command execution by avoiding DNS lookups.

Appending the | grep http pipe command filters the output of netstat for lines containing the word http, which indicates processes that are listening for incoming HTTP connections.

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[How to find your public IP]]> https://www.lambrospetrou.com/articles/public-ip/ public-ip Mon, 04 Dec 2023 00:00:00 GMT Open https://checkip.amazonaws.com/.

Use the following to get it programmatically:

curl https://checkip.amazonaws.com
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Template for writing technical RFC docs]]> https://www.lambrospetrou.com/articles/rfc-template/ rfc-template Sat, 18 Nov 2023 00:00:00 GMT Table of contents

A. Get the template

Download or copy the full template at this Google Document. Use it as you wish 😉

Once you open the template:

  • To use it with Google Docs: File (top left) > Make a copy
  • To use it with Microsoft Word: File (top left) > Download > Microsoft Word (.docx)

B. Sections overview

1. Title and reviewers

At the top of the template, you should have a clear title of the proposal. I usually prefix it with RFC - <title> to make it easier to search later in Google Drive.

After the title there is information about the state of the RFC, followed by required and optional reviewers status.

The state of the RFC can be one of: Draft, Under-review, Approved, In-progress, Completed

The first three states are used during the review of the RFC, and once approved we proceed to the next two states which refer to its implementation and rollout.

The list of reviewers should include:

  • Name or email of reviewer
  • Team name of reviewer
  • Review status: not-reviewed, in-progress, approved, declined
  • Review date (date when the final decision was taken)
RFC template title and reviewers section
RFC document title and reviewers section.

2. Table of Contents

I always include a table of contents to make it easy to jump to sections, and give an overview of what’s in the document.

Note: In Google Docs, make sure you “refresh” the table of contents widget after adding/removing sections in the document because it doesn’t auto-update. You can refresh it by clicking anywhere inside the table of contents, and clicking the “refresh” icon at its left side that will appear.

3. Overview and context

In this section you should describe the problem the RFC is addressing, and provide necessary context so that people unfamiliar with it can understand the benefits of implementing the RFC. There can be sub-sections to give more detailed information.

Glossary and terms (optional)

Provide some explanation of terms that will be used later in the doc.

Customer/Business impact (optional)

Provide evidence of customer impact or business impact that justify working on this RFC.

4. Goals and Requirements

Explicitly mention what is in-scope and out-of-scope for this RFC.

5. Timeline and Milestones

Provide rough estimates and track key milestones for the RFC and its implementation, including its rollout. This section is to be updated over time as the RFC progresses from review, to approval, to implementation, and to its rollout.

6. Proposal solution

High-level overview

This section describes the proposed solution. It should be enough for everyone to understand what the RFC is proposing, and how it’s solving the problem while satisfying the goals and the requirements mentioned above.

This section should not go into all the technical details (see relevant section below), since some stakeholders (e.g. product managers, directors, VPs) might not be technical. It should however provide enough details to be complete on its own, so it’s not just a vague description of a solution.

Put 1-2 high-level diagrams explaining the solution, but not too many, and describe the key components and the core process/flow of the solution.

Most of the non-technical folks can stop reading here. Maybe they can also read the questions section at the end, but they shouldn’t need to read the technical details section below.

Technical details

This is the meat of the RFC for the technical folks. There should be sub-sections for all key aspects of the proposed solution.

Each sub-section should cover one key component of the solution and give details to fully understand how it will be implemented. Include diagrams, code snippets, schema definitions, and decisions taken with their reasoning tradeoffs.

Open Questions (optional)

This section contains open questions about the technical implementation that are yet to be investigated or decided, but won’t materially change the proposed solution itself, so it doesn’t block its review and approval.

7. Alternative options (optional)

This section should give some details about alternative options that were researched and rejected. You might not need this section, if the proposed solution is straightforward or it doesn’t really have many different alternatives worth mentioning.

There can always be discussion and changes in the technical implementation in the previous section, but this section is for significantly different approaches to solving the original problem.

8. Frequently Asked Questions

This section should contain questions that you expect folks to ask, or are being asked a few times after publishing the RFC so that you don’t repeat yourself in comments, and can just point folks to this section.

I usually prewrite some questions I expect that colleagues will have, and populate it with more as people review the document.

9. Appendix (optional)

I rarely put an appendix into my RFCs because usually I see other folks abusing it by putting too many unnecessary details.

The appendix can be used for more detailed diagrams, screenshots, tables of data, and in general secondary information that supports the rest of the RFC.

Depending on how many implementation details you need in the RFC you usually don’t need to even have an appendix, since it leads to annoyingly long documents.

When you need it though, include it at the end of the document and keep it tidy with sub-sections.

C. Template preview

Download or copy the full template at this Google Document. Use it as you wish 😉


]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Amazon Profiler and AWS CodeGuru Profiler — CI/CD]]> https://www.lambrospetrou.com/articles/cicd-amazon-aws-codeguru-profiler/ cicd-amazon-aws-codeguru-profiler Sun, 22 Oct 2023 00:00:00 GMT Table of contents

In this article I will go into details of the CI/CD pipelines my team implemented while at Amazon. Read my previous article for “The CI/CD Flywheel” to understand how each of the pipeline segments we will explore maps to the software release lifecycle.

Important notice

Amazon is a huge company, with hundreds of engineering teams.

The examples showcased below are based on my own experience. However, you should keep in mind that each team, organisation, department, could follow different practices, sometimes in significant ways.

Anything described in this article should not be extrapolated to be how the whole company worked. As you will see, even within my teams, we had different processes for several things.

You can find a more generic description of Amazon/AWS deployment pipelines in these amazing articles in the Amazon Builders’ Library, both written by Clare Liguori:

Amazon Profiler

I was in the Amazon Profiler team for about three years before leaving Amazon in 2020. Our product was continuous CPU profiling [1] [2] for the internal backend services, mostly targeting JVM services.

In this team we were doing trunk-based development for all our services, and all Pull Requests were merged into a single mainline branch.

Let’s explore the CI/CD setup for one of our backend services, and our website.

Backend

One of our backend services was the Aggregator, processing all ingested profiling data submitted by all the services using our product, and generating aggregated data that we would then display on the website.

Diagram of the CI/CD pipeline for Aggregator service
CI/CD pipeline for Aggregator service.

We had one pipeline with 4 stages, including 3 deployment environments (beta, gamma, production). All the environments were in the US.

The source packages column is a collection of Git repositories that will trigger a pipeline execution on every push on the specified branches. Amazon doesn’t use monorepos, therefore we could have multiple repositories (we call them packages) per service.

The first stage, build, compiles the source code, runs a few static code analysis jobs, and builds the release artefacts. In our case, the artefacts were the Java .jar files, along with some configuration files.

Proceeding to the beta stage, these artefacts are deployed to the beta environment. This is a development environment with limited resources (single host), meant to be used only by our team. The goal of this environment is to guarantee functional correctness of our product, and of our automation. Stress testing, and high availability is not a goal for beta environments.

Once the deployment completes, we run approval workflows (e.g. integration tests) against the deployed environment, targeting its API directly.

To promote the release artefacts from the beta stage to gamma, we had a manual approval gate. The manual promotion is not always the default. We alternated between manual/automatic promotion depending on if we wanted to pause changes from being deployed. This was useful when we did manual testing in gamma for specific features or to troubleshoot issues.

The gamma stage is usually considered the last pre-production stage at Amazon. This environment is closer to production, configuration-wise, region-wise, etc. In some aspects though, gamma and production are not identical. Especially size-wise. It would be inefficient for all our gamma environments to have the same size as our production environments.

The idea is that gamma is where you do all the tests that need a production-like environment, without affecting actual customers. A common use-case for gamma stages was to run our load-testing tool, TPSGenerator.

In the case of our Aggregator service, we would mirror the traffic from production in this environment so that we have similar traffic patterns, and similar ingestion data, in order to run experiments on production-identical data and catch issues before impacting our customers.

The approval workflows in gamma are more rigorous than beta, and they included (not exhaustive list):

  • load testing
  • integration tests
  • long-running aggregation tests (e.g. multi hour aggregations)
  • security oriented tests

Once all the approval workflows passed, we were manually promoting the release artefacts to the production stage. A final round of approval workflows were running against the production environment, and if any workflow failed, an automatic rollback would trigger the deployment of the previous artefacts.

One common approval workflow we had across our services was the monitor bake time. For a specified amount of time, the pipeline would continuously evaluate a collection of PMET monitors (internal CloudWatch predecessor) and if any of them transitioned into Alert state within the specified period, the approval workflow would fail, automatically triggering a rollback to the previous release artefacts.

Note: We didn’t do rollbacks in beta, since it didn’t matter if it was broken for some time. It would only affect our team, and it would give us time to investigate and fix the offending commits. But, we did care for gamma and production.

Website

Our website was a Single-Page-Application using React. We had a simpler setup than the backend, using AWS services: CloudFront, S3, and Lambda@Edge.

Diagram for the CI/CD pipeline for Amazon Profiler website
CI/CD pipeline for Amazon Profiler website.

We had one pipeline with 3 stages, including 2 deployment environments (beta and production).

This is as simple as it can get, but incredibly sufficient for many teams.

This is what I call The Core pipeline, and everyone should try to have this as starting point.

The source packages column is a collection of Git repositories that will trigger a pipeline execution on every push on the specified branches.

The first stage compiles the source code, runs static code analysis jobs, and builds the release artefacts. For the website, the release artefacts were the JavaScript/HTML/CSS files we had to deploy in Amazon S3, along with some AWS CloudFormation templates for the deployment on AWS.

Proceeding to the beta stage, the artefacts are deployed to the beta environment, in AWS region us-east-1. The approval workflows run once the CloudFormation deployment is done, and verify that the website loads and functions properly using Cypress UI end-to-end tests.

We once again have a manual approval gate before promoting the artefacts to the production stage, same region.

Manual testing against backend API

The beta environment is identical to the production one in this case, since we were using AWS in exactly the same way. The only difference was the domain name.

We could point the website to use any of the backend environments with an in-app toggle. Within the beta website you could switch between the backend APIs of beta, gamma, production. We were using the beta environment of the website during development until we felt good to release in production.

Even though we had the manual promotion, we were releasing regularly to production, several times per week to avoid big-bang releases.

Preview environments

Oh the hype… All the modern Platform-as-a-Service products (Vercel, Netlify, Qovery, etc) brag about their automatic Preview Environments per PR.

Even though Amazon tooling didn’t provide this out of the box, it was easy to do it ourselves.

  1. We made sure that even when running locally on our laptops, we could target any of our backend APIs, in addition to the mocked responses locally.
  2. We setup one of our development boxes (remote EC2 machine we sometimes used for development), and before publishing a Pull-Request a convenient command would ship the artefacts onto that host and give it a unique URL that we put in the PR description.

We could have used separate AWS accounts/regions and deploy the full website per PR, but at the time we found it easier to just use a host for all the PRs.

AWS - Amazon CodeGuru Profiler

Amazon CodeGuru Profiler is the AWS service my team launched at re:Invent 2019.

At the time, the public product was a small subset of what the internal Amazon Profiler service was capable, and the underlying infrastructure was different. As explained above, Amazon Profiler was using our internal deployment platform (Apollo), and was deployed in only one US region.

CodeGuru Profiler was built from the ground-up on-top of AWS services, using Infrastructure as Code to automate as much as possible, and was deployed across many AWS regions.

There are a lot to cover about launching an AWS service, but in this section we focus only the CI/CD pipelines.

As I mentioned above, we were doing trunk-based development for all our services, and all Pull Requests were merged into a single mainline branch.

All our backend services, including the website, followed the same CI/CD architecture, therefore I will use a generic service in the example below for simplicity.

Diagram of the CI/CD pipeline for Amazon CodeGuru Profiler services
CI/CD pipeline for Amazon CodeGuru Profiler services.
(right-click and open in new tab for full preview)

Don’t get frighten by the big diagram 😅 We will go through it piece by piece.

The first half part of the pipeline, up to beta stage, is pretty identical to the showcase of Amazon Profiler, so I will just skip it, and focus on the new parts.

Waves

Every AWS service has to deploy in tens of regions across the world. A region is a physical location of several data centers.

There is a big variance in customer adoption among the regions, which is the dimension of “region size” I will use below.

This is exacerbated by the fact that some services are only available in a specific region. One example is AWS Lambda@Edge functions. Even though it deploys your code across many regions to run close to your customers, their own control plane (create and manage your functions) is exclusively in us-east-1. Unfortunately, this means that if you want to use such services you need to deploy part of your infrastructure in those regions.

It is also important to note that deploying one region after the other would take too long for a change to reach all regions.

AWS teams use a wave-based deployment approach where you group multiple regions and deploy to them in parallel, whilst the waves are deployed sequentially. This significantly reduces the blast radius of faulty releases, from automation and configuration issues, to buggy features.

Remember, practicing CI/CD for releasing your software is a way to make sure you ship fast, but also reliably, without breaking many customers.

There are a few different approaches in how you should group your regions, and in what order, but the usual wave-based setup is:

  1. A small region
  2. A medium region
  3. 3+ regions
  4. 5+ regions
  5. N+ regions

The exact size of each wave is up to each team, but the idea is that you start slow, and then with each wave you go faster, with more and bigger regions.

The initial slow part gives confidence that your changes are safe, there are no obvious configuration issues, and the business metrics are not dropping alarmingly. Then, you accelerate the deployment in order to rollout the changes to every customer.

Environments

It is common practice in AWS services to have the following deployment environments per region:

  • gamma: The last pre-production environment for automated, and manual, testing before shipping changes to customers.
  • onebox: Often a single host, but could be a small percentage of the overall production cluster of servers, or even a small percentage of AWS Lambda traffic.
  • production: The full production environment.

To fully benefit from the onebox deployment environment, we need to have proper segmentation of key metrics that we use for alarms and monitors between onebox and rest of production. Otherwise, it would be impossible to monitor the impact of a change on the onebox environment if the metrics are covering the whole production environment.

Onebox environments are optional, depending on the underlying infrastructure used. For example, when using AWS Lambda in combination with AWS CodeDeploy, you can achieve the same result without having a dedicated environment, since CodeDeploy allows you to do rolling deployments by percentages while monitoring alarms at the same time.

This is what we did in our API services, we used AWS Lambda, and we didn’t have onebox environments.

Pipeline promotions

With the above context in mind, let’s dive into the pipeline itself.

Once the release artefacts are verified in the approval workflows for the beta stage, they are then automatically promoted to gamma.

In gamma we do parallel deployment across all regions. These environments are not exposed in customers, and the reason we deploy to all regions is to make sure that our configuration, especially infrastructure automation, does not have hardcoded values, or making assumptions about region-specific properties.

After a deployment for a region completes, the corresponding approval workflows start running, even if other regions are still being deployed. Hopefully, all the approval workflows will pass, and the release artefacts will be ready for promotion in the first production wave, wave-1.

In all the promotions targeting a production environment, there are two usual conditions:

  • All approval workflows pass.
  • The time window blocker is disabled.

The time window blocker is a feature where we block automatic promotion of changes to the next stage during a specific time window. We use this for important holidays like Christmas and Black Friday, or for restricting deployments only during specific times to avoid breaking production outside the business hours of the on-call, e.g. deploy only during EU/London office hours.

Proceeding to the production wave deployments, we first deploy to the onebox environment, if there is one. The approval workflow for onebox is often lightweight, mostly monitoring specific metrics, and making sure our alarm monitors do not trigger for a specific amount of time.

Then, the changes are automatically promoted to the full production environment of each region of the wave.

Reminder, that the deployments within a wave are done in parallel.

The approval workflows for each region of the wave start as soon as the corresponding deployment completes. Approvals here include business metrics monitoring, UI end-to-end tests (for certain APIs, and the website), and making sure our alarm monitors do not trigger for a specific amount of time.

In some cases, there is a also a fixed time bake period, where the approval workflow for the wave is essentially paused before proceeding with promoting the release artefacts to the next wave. This is useful when we want to make sure enough time has passed, and enough customers were exposed to the changes released, which could surface issues not covered by the automated or manual tests, and no metric regressed right after the deployment. For example, some issues like memory leaks could become prevalent only after several hours.

The fixed time bake period is not a rule, and I have seen many teams not having them at all, or having them and skipping them unless there is a specific reason.

Once again, when the approval workflow passes and there is no time window blocker enabled, the release artefacts are promoted to the next wave, until all the waves are deployed.

Automatic rollbacks

In case any approval workflow fails in wave N, then the promotion to wave N+1 is blocked, and the offending regional environment rolls back to the previous good version.

This means that some regions within the same wave, could be running different versions of your software than others.

The next version that will flow through the pipeline will at some point arrive at wave N, redeploy to the included regions, and the approval workflows will rerun. If the approvals pass, then the new version will be promoted as usual to wave N+1. Therefore, all the regions within the wave will now run the same version.

Read more about automatic rollbacks in the article “Ensuring rollback safety during deployments”.

Multiple in-flight versions

My absolute favourite feature of the internal Amazon Pipelines service (also available in AWS CodePipeline) is that each stage of the pipeline runs independently from others. ❤️

This seems subtle initially, but in practice this allows parallel environment deployments of different versions!

For example, you could have v10 being currently deployed in the production environment, v11 in the gamma environment, and v15 in the beta environment, all at the same time in parallel.

This allows for amazing productivity, and velocity, shipping features often and reliably. You don’t worry about coordinating releases, or babysitting a single version throughout the whole pipeline before deploying the next one.

You merge your change, and then it will automatically (with the optional manual promotions occasionally) travel through the pipeline stages as fast as possible, depending on the approval workflows.

If your automated tests are sufficiently good, you can merge multiple times a day, and the pipeline will continuously ship those changes across all your environments automatically.

Conclusion

This was quite a long article, and there are still many things we didn’t cover around CI/CD techniques in the real-world.

I hope this gave a glipmse into how some of the principles and theory of Continuous Delivery (CI) and Continuous Deployment (CD) are applied in practice, and hopefully it gives you confidence to implement some of these techniques in your own team’s release process.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[The CI/CD Flywheel]]> https://www.lambrospetrou.com/articles/cicd-flywheel/ cicd-flywheel Sun, 15 Oct 2023 00:00:00 GMT Table of contents

In this article you will learn what CI/CD is, why you should use it, understand why it can be a competitive advantage for any team, and in follow-up articles I am going to describe how CI/CD is implemented in real-world teams I have been working with.

Introduction to the CI/CD flywheel

Let’s step through the full development and release lifecycle of releasing a traditional web application. We start by writing code, testing it, building it, shipping it to production, and making sure it works as we expect.

The full flow is what I call The CI/CD flywheel, and as we will see at the end, all the steps of the flywheel are what a CI/CD pipeline implements.

Product

It all starts by our customers, or potential customers, needing a feature.

This process involves discussing and coming up with a well-defined feature that we want to implement that will solve the customers’ issue.

Different teams follow very different methodologies for product decision making, so I am not going to get into that aspect here, but at the end of this step we end up with a bunch of engineering tasks that we want to implement.

The CI/CD Flywheel - user feature requests and bug reports

Local - Write the code

Once the feature requirements are defined, we write the code.

We do either locally on our laptop or PC, or in a remote development environment. We use an IDE like Visual Studio Code, or a terminal based editor like vim.

At the end of the day we end up with some source code supposedly implementing the agreed-upon feature in our product.

The CI/CD Flywheel - locally write code

Local - Build and Test

Once we have some code written, we want to test it to make sure it does what we expect.

Usually, we write some unit tests to test at the function or class level, maybe some integration tests to test a bunch of things together, and if we are feeling adventurous we add some end-to-end (E2E) tests. Most probably, if possible, you run the application locally for manual testing to make sure it does what you expect.

We repeat the process of writing and testing the code until the feature is ready.

The CI/CD Flywheel - locally build and test

Remote - Review and Test

Once we are confident and satisfied with our code, we submit it for review by opening a Pull Request (PR).

Don’t worry, I know some of you just push directly without peer-reviews, but we are not all adrenaline-junkies…

Most (good) code review systems (e.g. Github) allow you to run several jobs on each Pull-Request (PR) to build and test the changes. In these jobs we run any kind of tests we have, and even do more elaborate tests that take more time to complete, and therefore we don’t want to be running locally every time we change something in our code that would hinder our productivity.

Ideally, these jobs run in isolated environments that do not interfere with any production system, and can provide reproducibility in every run.

In some other code reviews systems, we are not able to do much other than building the code, and running the basic tests (unit, integration), so we just do that. Either way, we want to take advantage of this functionality to do testing of our code changes before merging them to the main branches used by everyone else in the team/company.

The CI/CD Flywheel - code review and pull-requests

Remote - Merged, Integrated, Built

After the code review is done, we merge our changes into the main branch(es) we use for production releases.

Once again, we want to run tests to make sure the code is correct and properly “integrated”. Keep in mind that at this point potentially several other engineers have also merged their changes into the same branch.

The term “integrated” here means that we bring changes from multiple engineers and integrate them all together into one source, that needs to be built, tested, and then released as one. Even if there are no code-related conflicts, there could be logic/business oriented conflicts and issues because of this merging.

At this step, we run extra analysis jobs like static code analysis, or security vulnerability checks, or end-to-end tests that could take several minutes or hours to complete, and more advanced tests.

This is often the last step we will run most of our suite of automated tests without direct customer or manual interaction.

The CI/CD Flywheel - build and test

Remote - Prepare the release artefacts

Once everything passes, we need to build the release artefacts that will get deployed into our actual running systems.

Depending on the application this could be platform-specific artefacts (e.g. .apk) that we will upload to the mobile app stores (Apple Store, Google Play Store). It could be Docker containers that we publish to one of the container registries (e.g. Dockerhub, AWS ECR), or the binaries for our CLIs, or the static files for our website, or any other artefact that our build process generates.

Regardless of the type of our product, this is the step where we generate the artefacts to be delivered to the production systems.

The CI/CD Flywheel - release artefacts

Deploy - staging / preproduction

We wrote our code, we built it, we tested it, and we generated the release artefacts. It’s time now to deploy it.

At this step we take the release artefacts from the previous step and ship them to wherever our product is released.

If it’s the app stores, we will follow their corresponding process. If it’s a website or a backend server we will deploy the artefacts to the servers. If it’s a firmware update we will deploy it to some device lab with actual devices and automatically install it on them.

Finally, there are some rare cases where the deployment will happen in a manual way, by an actual person, to a medium that we cannot reach with automation and requires physical presence.

A key aspect for this step is that we should do the deployment on an environment that is not the one used by our customers, or at least not all of them at once by using private beta programs with few testers only (e.g. in the case of a mobile app). This is what we call staging or pre-production environment(s).

This is not always possible, and some teams prefer to just ship to production (if you are one of those - at least use feature flags!). Personally, I always prefer having at least one staging environment.

The CI/CD Flywheel - staging deploy

Deploy - staging / preproduction verification

We deployed our changes to our staging/preproduction environment, and it’s time to verify that no alerts or alarms are going off, and that all the features we implemented work as expected, old and new.

This verification could be done in various ways, automated, manual, in a few seconds, or across several hours. It all depends on the actual product.

The CI/CD Flywheel - staging verification

Deploy - production

Finally, we deploy our artefacts to the production systems used by our actual customers.

This process should be similar to deploying to our staging environment(s), albeit in a different scale and ideally in a gradual way across all of your infrastructure.

The CI/CD Flywheel - production deploy

Deploy - production verification

For one last time, we need to verify that everything works fine, no customer is negatively impacted by the newly shipped changes, and that we didn’t break anything that was previously working.

To succeed in this step we need to have proper monitoring and observability of our applications. There is a whole bunch of things around observability but at the end of the day, you should have alerts that will notify you when something is negatively impacting your users.

The CI/CD Flywheel - production verification

Customer feedback / New feature requests

Our changes are in production, used and loved by our customers, so now we repeat the whole process again. Customers are asking for more features, or they complain about something and we need to fix.

We go back to our editor, write some more code, and go through the whole cycle again.

The complete CI/CD flywheel

The CI/CD Flywheel - complete

The whole process we went through is what I call The CI/CD flywheel.

In traditional publications, it is also called Software Development Life Cycle (SDLC), but I like flywheel better because it emphasizes the looping aspect.

We can see that the whole development cycle, the idea/customer request, writing code, delivering an artefact to customers, getting feedback, and going back to ideas, is a simple loop. The faster you transition from one step to the next, the faster you deliver value to customers.

One way to cycle through the flywheel faster, is to remove certain steps, for example the verification steps, or even the whole preproduction environment. Indeed, this will shorten the circle. However, there is a hidden trap in this thinking.

Removing validation steps will cause you to actually go through extra iterations. Inevitably, broken things will be shipped to customers, and we will need to repeat the process more times just to fix bugs that could be prevented in the first place.

Catching issues later in the process is much more expensive than catching them in an earlier step. The latter steps in the flywheel (deploying, verifying and getting user feedback) are often more time-consuming than executing the earlier steps (writing and testing code), therefore doing the early steps more times is more beneficial than doing the latter steps more times.

In the diagram below we can see how the steps of the flywheel map to the CI/CD phases: Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment (CD).

The CI/CD Flywheel - complete cicd

Most of the flywheel belongs into at least one phase of a CI/CD pipeline.

Therefore, having great CI/CD systems is crucial in order to allow us to move faster, and delivering value to our customers safely, reliably, and continuously.

Why do we need CI/CD

We have explored the CI/CD Flywheel that gives an overview of the CI/CD process. Let’s explore some of the concrete benefits we can get from implementing CI/CD in practice.

Business benefits

Ship value to customers more often

Implementing new features, fixing bugs and issues detected in early stages of the CI/CD flywheel, and automated delivery of the release artefacts to the different environments, all ultimately contribute to shipping business value to customers faster and more often.

Earn and retain customer trust

With a variety of automated tests, early detection and fixing of broken features, gradual rollout of new features, and finally, automatic detection of faulty releases, we significantly reduce the risk of breaking features that customers depend on. Or, when we break features, we can quickly release their fixes.

In this way, we earn the trust from customers, and retain it forever reducing unsatisfied customer churn.

Fast experimentation feedback loop

Practicing CI/CD efficiently means that we can experiment and test new innovative features faster and more often.

Engineers are not being slowed down in cumbersome manual release procedures, and most importantly they are not risking in breaking real customers with unpolished or unfinished features unexpectedly.

CI/CD enables the development of features that would otherwise take a lot of back-and-forth in discussions among the different stakeholders until its decided that it’s worth doing.

Within a few days or even hours, a feature can be tested in a subset of users, or even in a staging environment without impacting customers, and take a more informative decision based on real data.

Attract the right talent

Anyone in a creative role, including software engineers, designers, product owners, and others, likes to see their ideas and creation used by customers as soon as possible.

The rapid development feedback loop that CI/CD provides is a great selling point to attract people into working for a company.

Who wouldn’t want to work on a team that allows them to work and ship new ideas every day, straight to customers, in minutes or hours, instead of months!

Cost reduction

The earlier an issue is detected, and fixed, the less costly it is to the business. Let’s examine some scenarios.

Scenario A - reckless

  1. We implement a new feature, and ship it directly to production so customers start using it.
  2. Customers start complaining in social media, and file support tickets, because something broke.
  3. The support team and on-call engineers are scrambling to see what’s broken, implement a fix, and then ship it straight to production again.

Scenario B - half-assed

  1. We implement a new feature, and ship it to an environment where a team of Quality Acceptance engineers (QAs) will need to do manual testing.
  2. After a few hours, or days, the QA team managed to test our changes but detected something is broken.
  3. They send it back to the engineering team, they implement a fix, and they send it back to QA again.
  4. After a few hours, or days, the QAs approve, and we now ship the change to production.

Scenario C - better

  1. We implement a new feature, and merge it in the main codebase.
  2. The change goes into the CI/CD pipeline where the first step is to run some automated tests.
  3. One of the automated tests fails and stops the pipeline.
  4. We see that the automated test suite failed, check which test failed, implement a fix, and merge again.
  5. The tests pass, and the change is delivered to our staging environments for further validation and then goes to production.

There are thousands of other scenarios that could happen in practice, but I chose these because they showcase well the difference in mentality, and more importantly the cost of fixing an issue.

In scenario A, the issue went straight into production, and customers noticed immediately.

This not only hurts the business reputation, but it also costs a lot more in person-hours.

We will spend a lot of time debugging the customer complaints until we figure out the root cause, then implementing the fix, and shipping to production. If we are lucky, the change we did for the fix won’t break something else…

In scenario B, we are not as reckless as in scenario A, and we have a team of dedicated QAs that will do manual validation of changes.

This sounds very inefficient (and it is), but it is actually used by thousands of companies, even today. The QAs will often catch issues before reaching production.

However, the issue is the overhead of communication between the engineers and the QAs, in some cases with multiple back-and-forth iterations. In addition, a code change just sits in a queue of changes that the QAs gradually go through and verify, which also introduces extra delays.

Finally, imagine all this happening twice. This is a lot of man-hours just spent on verifying that a change works as expected.

In scenario C, we spent a few minutes and automated (most of) the tests that the QA team would do manually in scenario B, and added them in the automated test suite that runs in our CI/CD pipeline.

The tests ran and failed just a few minutes after merging our change, and within a few minutes we implement a fix and merge again.

So, let’s recap what each scenario cost the business:

  • Scenario A: X hours of debugging and going through customer complaints & business reputation risk.
  • Scenario B: Y hours/days of developers and QA communication, but less risk of the issue reaching customers.
  • Scenario C: Z minutes of automated run on a remote machine, and less risk of the issue reaching customers.

In majority of cases, it holds that X ~= Y >> Z, which means that even though initially it seems that having a full CI/CD pipeline is more work and more steps, in practice it will save us a lot of time.

I hope it’s clear how big of a cost reduction it is to detect issues early on and to automate their verification.

Fun fact 👉🏻 CI/CD is similar to Anton Cord, introduced by Toyota. The farthest the issue was detected in the pipeline the costlier it was to fix and it would cause more defects. The same approach was adopted by tech companies like Amazon and Netflix, and then more.

Technical benefits

Code quality

Due to integrating changes to the main source code repository multiple times per day, we guarantee that the changes always build, and pass the relevant tests.

Gone are the days that engineers spent weeks implementing features in isolation, and then spending days trying to merge their changes together, solving major conflicts, or even rewriting big chunks of code to make it work.

Comprehensive tests

Because our CI/CD pipeline will run automated tests for each code change in remote environments, we are encouraged to add more tests that will cover more scenarios we want to check.

We can have tests that need elaborate setups, and that would be very time consuming to setup manually, like privacy and security checks, or performance regression detection, or testing our change on multiple versions of a browser, or multiple devices, etc.

Overall, due to having automated test suites running on isolated environments we can do things we cannot afford to manually do locally on every code change.

Automate repetitive tasks for reproducibility

Doing manual steps over and over again inevitably leads to mistakes.

QA testing sessions usually include manual actions. Publishing release artefacts by manually running commands locally on your laptop is problematic.

I faced such issues early on in my career. For the release of our mobile app we were using our own laptops to build the corresponding artefacts for each app store (iOS, Android, Amazon Fire). Due to differences in our workspaces, once or twice, we shipped faulty releases. However, when a different engineer was trying to reproduce the issue on their laptop, it wasn’t failing, due to local workspace differences.

This was a nightmare to debug.

A good CI/CD process will always guarantee that everything runs from a clean state, so that every time you execute the same steps, you get the same result.

Automating as much as possible is a gift that will never stop giving! 🎉

Build once, use everywhere

When using CI/CD to build and deploy our software, we get the nice benefit that the same artefacts we build in the first steps of the pipeline are the ones being propagated to all the environments thereafter.

This means that we can attribute anything that happens in the pipeline, like test failures, or deployment issues, or even issues caught in production, to a single artefact, which ultimately will lead us to the offending code changes easier and faster.

Another benefit from releasing the same artefact everywhere, is that we can reason more easily about what’s running in each environment/stage without worrying about inconsistencies among the environments.

Controlled rollouts and faster Mean Time To Resolution (MTTR)

Automatic deployments across environments enables the use of controlled, gradual, rollouts.

We could be doing gradual rollouts on a small percentage of customers each time, or even rolling out changes continuously but hiding the feature behind feature flags and only enabling it for certain customers.

This allows monitoring to detect failures impacting customers, and rolling back the newly released changes automatically. This is easy and efficient to do in a CI/CD managed environment since each change that gets released is usually small and its rollback won’t cause havoc to customers.

Doing this manually, even if doable in some cases, would be very error-prone with elaborate manual steps.

Real world CI/CD

In follow-up articles I will provide a bird’s eye view of some real-world CI/CD systems and procedures that I personally experienced throughout my career.

I want to showcase some examples of how companies use CI/CD in different ways, and at the same time how they follow the same principles that make up the CI/CD flywheel!

In all these real-world system the fundamentals are the same. The goals are the same. The benefits are similar. The implementation is different.

CI/CD systems we will explore:

  • Amazon Retail / LOVEFiLM
    • Gitflow development (2 branches), 2-3 stages (beta, gamma, prod) and 2 regions
  • Amazon Profiler
    • Trunk based development, 2 stages (beta, prod) and 1 region
  • AWS/Amazon CodeGuru Profiler
    • Trunk based development, multi stage (beta, gamma, prod), multi region
  • Meta WWW
    • Trunk based development, tiered production deployment (C1, C2, C3, … C9)
  • Others…

If you want to see something specific please reach out @lambrospetrou 😅

Conclusion

This article introduces the CI/CD flywheel, and maps its steps to a CI/CD pipeline for building, testing, deploying, and verifying our software products.

I truly believe that embracing and implementing fast automated CI/CD pipelines is truly a competitive advantage for any team, and any company, of any size! 💪🏼 🚀

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Writing docs]]> https://www.lambrospetrou.com/articles/writing-docs/ writing-docs Sun, 24 Sep 2023 00:00:00 GMT Table of contents

Context

what’s your motivation to create such good docs?

A couple of days ago, a colleague asked me the above question. That was the 4th person commenting positively within the same week about that specific wiki I wrote, so I decided to write down an article about my thinking.

A bit of background, to set the stage right.

I personally love writing detailed documentation about the projects I do.

By “writing docs” I refer to:

  • Commenting code with explanation and reasoning for things that are not obvious.
  • Putting plenty of information in issue tracking tools (e.g. Jira, Github Issues), and pull-request descriptions, such that there is clear trail of what I researched, what I did (even if it didn’t work out), and what’s remaining.
  • Writing comprehensive wikis and runbooks for the bigger type of projects I work on.

This article is not an exchaustive list of benefits for having good documentation. Plenty of longer posts do that, and thousands of debates took place regarding source code comments, in favor and against.

The rest of the article enumerates the key reasons why I like writing docs, and hopefully they will convince you to do so as well.

Future me will thank me

Reason #1 is myself. How selfish am I, right?

I first and foremost, write these docs for myself. After a few weeks of not working on a project, I always forget things.

Such things include:

  • System architecture of the service, which services it interacts with upstream/downstream, and what are the key properties of the service.
  • Key project/service references to documents, issues/tasks, dashboards, oncall handles.
  • How do certain intricacies of the system work? Wiki sub-pages for important topics that I will almost definitely forget in 6 months, and I will need to remember when debugging a service at 03:00 in the morning after getting paged.
  • Common operations like building and deploying the service artifacts. I know we all wish that all services had a single command, and that command was the same across all codebases. Unfortunately in the real world things are hairy. Jumping from project to project, in different languages, and using different tooling makes it harder to remember everything. Give me copy-pasteable commands any day of the week!

There are many more things that I document, but these should already make it clear that there are plenty of things you need as an engineer that you will inevitably forget after a few months.

Having detailed and clear step-by-step docs is the easiest way for me to ramp up again on the project’s main aspects when coming back to it after a while.

Onboarding new joiners

No matter how junior or senior someone is, when you join a new team or an existing long-running project you have to spend a lot of time building up context.

There is a lot of tribal knowledge that someone working on a project builds up, and someone new joining the project doesn’t have. It’s awesome when you have nice colleagues that can answer all your questions promptly, but we are not always so lucky.

Many projects are single-person projects, so even though a team has several folks, only that person knows most of the project’s specifics. If that person is unavailable, good luck to you 😅

I witnessed a project where the main engineer was on vacation for a few weeks, and nobody else knew much about the project, so the project was put on-hold until they returned. To me, this is a huge no-no, and a red-flag!

This is common, and works fine in scrappy small startups where each engineer is essentially a department on their own, but as you grow, and as more folks join a team, it’s important to increase the bus factor of the team.

I personally treat documentation as my always-available assistant to explain things and provide handy tips. Thus, I always put effort in keeping docs updated.

Even basic operations like compiling code, running tests, and which dashboards to explore, are a great initial boost.

Having nice runbooks makes the onboarding process smoother, and you can focus more on the core parts of your onboarding, eg. understanding the business aspect of the product.

Asked twice - asked often

When I work on a project where more than 2-3 people are involved on a daily basis, it’s almost certain that every few days someone will ask the same question as someone else did before.

Some will even ask the same thing multiple times spread out across days. People forget easily.

My strategy is that if someone asks me the same thing twice, or I get the same question from two different people, I quickly write it down in our wiki so that the next time someone asks, I can just deeplink them to the wiki entry.

This not only wastes less time in the future by not typing the answer over and over again, but it also pushes more people into the wiki.

My grand-plan is that gradually folks will put the wiki into their own daily workflow as well. They will use it first to search for what they want, and ideally they will also start contributing to it their own tribal knowledge.

Even if I turn 1 person per month to improve our wiki, I consider it a win.

Conclusion

The main drawback of writing docs is that someone spends time doing it. But, in my experience, once you make wiki-writing just another item in your daily workflow, it becomes invisible. It’s not a chore that you do separately, you do it continuously in small amounts.

Taking notes and writing detailed docs is my superpower.

I use my personal notes tens of times per day, and I use internal wikis as my entrypoints to most things. You should start too.

If you want to read more about my personal notes, and the format I use, checkout the following articles I wrote in the past:

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Feature Flags — CI/CD]]> https://www.lambrospetrou.com/articles/cicd-feature-flags/ cicd-feature-flags Sun, 17 Sep 2023 00:00:00 GMT I originally sent part of this article to the mailing list of my Elements of CI/CD course, and now sharing it with the rest of you with improvements from the received feedback.

Table of contents

Overview

You should adopt and use feature flags as much as possible!

Feature flags allow you to deploy changes to your software rapidly, and continuously, as many times a day as you want, without sacrificing the reliability of your product.

When you use feature flags, deploying changes is only half the story, since no customer will be exposed to those changes by default. The second half of the story, is that you selectively enable features to a subset of your customers. This sounds simple, but it really is a superpower! 💪🏼

Imagine that your product is an Android mobile application where users can search for whiskies, and ultimately buy bottles directly from the website. You are developing two new features that each needs a few weeks of development and iteration.

  • Feature A is getting whisky recommendations based on whiskies you mark as liked.
  • Feature B is the addition of new payment providers that would allow customers to buy with less fees.

Two different features, developed in parallel by different (or same) engineers.

In some teams, you would work on these features on separate git branches, and only merge on the production branch that gets deployed when the feature is ready to go live. This means that you will have two separate branches diverging more and more over time, and you also don’t know if something else breaks until you merge back to the production branch.

Too late.

With feature flags, you can continuously merge your code changes to the production branch, and more importantly deploy them to production, while at the same time ensuring that nobody will get exposed to them before they are ready.

We do this by wrapping these in-progress/unfinished features with special conditions that check if the feature is enabled, and if yes, for which users, and only proceed if the current user is allowlisted for the feature.

In our whisky application above, let’s assume that feature A will not be enabled at all, and feature B will only be enabled for customers in England since the payment providers we implemented so far and are ready to be tested are for English customers.

Your codebase would call the following Java snippet when displaying the checkout screen:

record User(String id, String country) {};

List<PaymentProvider> getPaymentProviders(
  User user, 
  FeatureFlags ff) { 
  List<PaymentProvider> providers = new ArrayList();
  // ... some code that adds payment providers already supported

  if (ff.isEnabled(FeatureName.PayProviderXYZEnabled, user)) {
    providers.add(new PaymentProviderXYZ());
  }

  return providers;
}

The above snippet ensures that the PayProviderXYZForEngland is included in the checkout only when the feature flag FeatureName.PayProviderXYZEnabled is enabled for the user being handled.

A partial implementation of the FeatureFlags class for our whisky application can be the snippet below:

record FeatureFlag(
  Set<String> countries, 
  Set<String> userIds, 
  boolean enabledForAll, 
  boolean disabledForAll) {};

class FeatureFlags {
  enum FeatureName {
    PayProviderXYZEnabled,
    WhiskyRecommendationsEnabled
  }

  Map<FeatureName, FeatureFlag> rules = ImmutableMap.of(
    FeatureName.PayProviderXYZEnabled, new FeatureFlag(
      ImmutableSet.of("England"), 
      Collections.emptySet(), 
      false, 
      false
    ),
    FeatureName.WhiskyRecommendationsEnabled, new FeatureFlag(
      Collections.emptySet(), 
      Collections.emptySet(), 
      false, 
      false
    )
  );

  boolean isEnabled(FeatureName featureName, User user) { 
    FeatureFlag ff = this.rules.get(featureName);
    if (ff.disabledForAll()) {
      return false;
    }
    return ff.enabledForAll() || 
           ff.countries().contains(user.country()) || 
           ff.userIds().contains(user.id());
  }
}

In the above implementation of FeatureFlags our features can be selectively enabled for users based on their country, and their ID.

I have seen teams and companies implementing feature flags in vastly different ways. There are implementations as simple as the above, and there are implementations that have complex rules as we will explore in following sections.

You can see that in a few lines we have a functioning feature flags system that allows you to selectively execute code based on the user, or any other condition you need. This means that you can safely merge your code changes even if the features you work on are unfinished, or even incorrect, as long as you wrap their entry point execution call with a feature flag condition check.

Feature flags are a superpower

Feature flags are a superpower, and by using them you:

  • Can continuously merge and deploy changes, thus increasing your velocity.
  • Can make sure that all the in-progress changes are integrated into the production branch and do not cause other features to break.
  • Avoid maintaining separate diverging git feature branches that go out of date and lead to annoying time-wasting merge conflicts.
  • Can test features and get early feedback with as few, and as many, customers as you want.
  • Can quickly enable, and more importantly, disable, a feature by just changing a boolean value. Quickly rolling back broken features is now trivial!

The rest of the article will focus on popular use-cases for using feature flags. Going through these use-cases will hopefully make the benefits and power of feature flags clear.

Also known as

Feature flags also go by the following terms (non-exhaustive):

  • Feature switches
  • Feature toggles
  • Experiments
  • A/B testing (feature flags can be used for A/B testing as well)

Dynamic configuration

Feature flags are one use-case of dynamic configuration in our software applications.

In the code snippets above implementing the FeatureFlags class for our imaginary Android application, we used hardcoded rules for the features we wanted to conditionally enable.

This means that in order to update these rules, e.g. adding new user IDs to existing flags or adding new flags, we would need to go through the application CI/CD pipeline, and deploy the application itself to use the updated rules. Depending on the nature of our product this might not allow us to iterate on the feature flags themselves easily, and we wouldn’t be able to rollout many changes per minute/hour.

To solve this issue, we need to move the definition of rules outside our main applications, into their own artefacts, with their own CI/CD pipeline that can be executed independently of the application’s pipelines.

A common approach I have seen in practice is to put the feature rules in text files (e.g. JSON, YAML, TOML) and have a CD/CD pipeline deploy them in Amazon S3-compatible buckets in different regions.

Our applications will have to be adapted to periodically (every 1 minute) fetch these configuration files, and recreate the rules inside the FeatureFlags class based on the latest configuration files. This exact flow was how one of our feature flag systems worked a few years ago, while I was working on the AWS Console for Amazon CodeGuru Profiler.

An alternative would be to use SaaS services like LaunchDarkly or PostHog feature flags to configure your feature rules, and then in your application code you would call their APIs to get a decision.

There are myriad ways you can make your feature flags dynamically configured, but in all cases you want to have:

  • Separate deployment pipeline for the feature rules, outside the application’s deployment process.
  • Quick rollout of feature rules updates (forward updates, and rollbacks).
  • Periodic update of the active feature rules inside the applications.
  • Safe rollout of feature rules (if latest version is invalid, use the previous one to avoid breaking the applications).
Diagram showing the flow of dynamically updating feature flags and propagating them to the application with S3
Example of data flow for updating, storing, and fetching feature flag rules from Amazon S3.

Not just for the server

Feature flag systems are not only for servers and backends.

Even though the actual feature rules will indeed have to be served by some server API, at some point, they can be used for multiple application types.

You can use them on websites.

When the static assets are served, you can inject the feature rules inside the HTML document. Or, you can provide a dedicated API that the website will call once loaded to fetch all enabled features for the user session, and update periodically. Or, you can retrieve the feature rules every time the user logins or refreshes their session token.

You can use them on mobile applications.

Hardcoded feature rules in the application itself can be used, but will only be able to be updated from inside the application itself (e.g. user enabling an experimental feature), or when the application itself is upgraded. Usually, you provide a dedicated API that the application will call once loaded to fetch all enabled features for the user, and repeat periodically.

Offline applications.

In cases where there is no network connectivity or calling remote APIs is not possible, we can still use feature flags in the hardcoded fashion we explored previously.

The user of the application will need to do specific actions to enable or disable the features. For example, in several CLI applications you need to provide specific commands to enable experimental features (e.g. Node.js). In Android systems, you can enable advanced Developer Mode by clicking a specific menu item X number of times.

In general, with a bit of imagination you can use feature flags in any kind of software application we build. And you should.

Beta testing and allowlisting

One important aspect of agile product development is to get feedback from customers as early as possible, and iterate over the application by fixing issues and implementing new features.

Feature flags allow us to expose incomplete and in-progress features to a subset of our customers.

This makes it easy to iterate on our applications without worrying that we are going to negatively impact the rest of our customers. Not only that, but we can do it straight in our production environments, using real data, real dependencies, and real customers. The feedback and confidence we can get by using production directly is great.

Feature flags are a great way to implement early access programs for your products, and even paid tester programs where you allow organisations and individuals to test your product, and give you feedback, before releasing it to the public.

Rule conditions

In the example implementation for FeatureFlags we used the user’s ID and originating country as the rule conditions. There are hundreds of different conditions that we can use.

Simple user conditions include:

  • ID
  • originating country
  • pricing plan
  • device type
  • company/organization of the user (in B2B cases)

Apart from user specific conditions, we can have global conditions:

  • deployment environment region (e.g. only in AWS us-east-2)
  • percentage of our hosts (e.g. only 10% of our hosts should use a feature)
  • percentage of users (e.g. only 10% of the users should use a feature)

These are just few conditions I have used in the different teams I worked with, and there are way more. You can even combine multiple conditions together to get fine-grained control of your features.

Feature rollout

Probably the most popular use-case of feature flags is to gradually, and safely, rollout a new feature.

  • While the feature is implemented the feature is enabled only for certain internal users (or nobody), and only in our staging environments.
  • Once it’s ready for beta testing we enable it for a few customers, and for specific production environments.
  • Once it’s ready for full release, we enable it for each of our production environments, ideally not all-at-once to avoid breaking all customers in case a bug slipped through.
  • If the rollout completed successfully, we remove the code that does the condition check and always use the newly launched feature.
  • If the rollout of the feature caused some regression, we can update the rules to disable the feature, and deploy that in order to quickly disable the feature and revert to the working version of the application.

This is the most basic, most common, and arguably the most important use of feature flags.

Deliver value to your customers, safely, reliably, continuously!

AB testing and experiments

A more complex use-case for feature flags is A/B testing (experiments).

A/B testing is when we want to experiment with different variations of the same feature, for example choosing the color of the checkout button between yellow, green, blue. A/B testing systems usually provide extra features on-top of what we already explored so far, but the underlying technology is often the same.

For example, at Amazon, for the retail website we had our own internal service for feature flags and experiments called Weblab [1] [2] [3].

In Weblab you could create a new feature, or experiment, where you didn’t just specify the rules that enabled a feature. You could specify multiple treatments of the feature, and the rules per treatment.

For example, for the checkout button color example, you would have the Control (C) treatment, which is the default/existing case when the feature is disabled. Treatment T1 was the first option of the feature, e.g. yellow button. Treatment T2 was the second option of the feature, e.g. green button. Treatment T3 was the third option of the feature, e.g. blue button.

If you just wanted the feature flag functionality you were done, and in the code you would have something like below (not real Amazon-code, just as example):

String treatment = flags.getTreatment(FeatureName.ButtonColor, user);
if (treatment == "C") { 
  this.buttonColor = this.colorDefault; 
} else if (treatment == "T1") {
  this.buttonColor = this.colorYellow;
} else if (treatment == "T2") {
  this.buttonColor = this.colorGreen;
} else if (treatment == "T3") { 
  this.buttonColor = this.colorBlue;
}

If you wanted the A/B testing (experimentation) functionality, Weblab would also track key metrics that you specified for each of the treatments.

In the example above, we could track the number of button clicks for each treatment, and therefore we would be able to get concrete data on which button color performed better.

The Weblab implementation showcases that when there is a robust feature flag implemented, there is a lot of interesting functionality that is now made possible.

Kill-switch

Another common use of feature flags is the kill-switch.

In many important feature rollouts, and big events (e.g. Black Friday, Superbowl, Christmas), you might want to have an easy way to enable/disable specific functionality in your application quickly. In the kill-switch use-case, we want to disable a feature, or immediately revert to a different implementation of a feature.

Having a kill-switch is very similar to the rollout of a feature that went wrong and we disable it until fixed. But, instead of the feature flag being temporary, it’s permanent.

One real-world example of a kill-switch feature flag I encountered was in Amazon Video. When we released super popular shows (e.g. The Grand Tour) the demand was very high. We had several kill-switches all over the codebase in order to quickly disable certain functionality that would allow the services to scale better if we had unexpectedly high traffic.

For example, one of those kill-switches would switch from using server-rendering for the TV Show details page on the Amazon Video website, to using a static file from our CDN that would only show static information and allow you to stream the show. Even though some features wouldn’t be provided (e.g. customer reviews), this emergency measure would allow customers to watch the show, which was the main goal.

In general, most feature flags are meant to be temporary to control the rollout of new features. There are certain cases where we want the ability to dynamically change our application’s behavior, and that’s where permanent feature flags come in place, the kill-switches.

The deactivation/killing of certain features to allow others to perform better, falls into the graceful degradation technique (see AWS reliability pillar, see META’s Defcon) we can employ in our complex systems in order to cope with scale.

Testing feature flags

A common question I get from folks when discussing about feature flags is how to approach testing.

For every treatment of a feature flag, you essentially create another branch of logic that needs to be tested. Do your unit/integrations tests need to to cover all possible values of the feature flag, or not?

Personally, I do the following thinking process:

  • If some code is irrelevant to the feature flag, and is not affected by what the feature flag value will be, then I don’t amend the tests related to that code.
  • If some code is directly by the value of the feature flag, then I will try to cover it with tests.
    1. If the tests can become parametric, with the feature flag value as input parameter, the best would be to run all those tests for all the possible feature flag values.
    2. If the tests cannot become parametric, then I usually test the happy path for all the feature flag values, and then some edge cases for the final feature flag value.
    • For example, if we introduce a new feature flag for some new functionality, all the existing tests should still pass!
    • Once I confirm that they pass, I mock the feature flag value for those tests to be what the final value will be after the full rollout. This will make sure that existing functionality will work fine once the feature flag is fully rolled out.
    • Finally, I will duplicate some important tests (or somehow make them parametric) with the feature flag off to make sure that some new logic will not inadvertedly depend on the feature flag, and in case we roll back its release we end up with issues.

Testing on its own is a hot-topic. Some engineers are neutral about it, others like it, others hate it, and you have the extreme camps on either side.

I personally like to have some tests. I definitely don’t push for arbitrary 90%+ coverage, but I also don’t like having zero tests for core business logic because it makes development much much slower.

If you like tests, then make them cover the feature flags too. If you don’t like tests, then you shouldn’t even be reading this section.

Managing feature flags

Managing feature flags includes all the actions and procedures you need to have in-place in order to boost your productivity while doing safe rollouts of new features.

  1. Have one defined process everyone will follow when creating, updating, and deleting feature flags. This process can be automated with simple CLIs, any low-code workflow tools that can interact with source control, or using any of the SaaS services dedicated to feature flags.
  2. A feature flag should almost always have the following lifecycle:
    1. Creation and code changes introducing branching behavior.
    2. Ship code changes with feature flag OFF in staging/production.
    3. Gradually enable the feature flag ON.
    4. Once the feature flag is fully enabled, monitor key metrics for N days/weeks.
    5. Once the verification is done, and the feature flag has been fully enabled for sufficient time, you should now delete the code changes using the feature flag, and make the enabled code branch the new default!
    6. Delete the feature flag.
  3. Do code reviews for any feature flag change. Enabling or disabling a feature flag, changes the running code, so treat it as any other code change.
  4. Implement some kind of monitoring to notify you when a feature flag is fully enabled for N weeks, and not yet cleaned up. Or even worse, when a feature flag is partially rolled out for N weeks without new changes.

As I said above, a feature flag introduces branching in the code behavior, so you should try to clean them up as soon as possible.

Leftover feature flags, especially partially rolled out ones, are a nightmare to maintain.

Conclusion

Use feature flags everywhere! Backend servers, frontend websites, mobile apps.

Iterating on new features development without the fear of breaking production and impacting customers is a productivity booster.

You can start by a simple mechanism of hardcoded feature rules in a file, periodically fetched by your servers, and expand into more complex solutions later.

Deliver value to your customers, safely, reliably, continuously! 🚀

Changelog

  • 2023-09-24
    • Added sections “Testing feature flags” and “Managing feature flags”.
    • Added table of contents.
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[How to pass the interview for software engineering roles in Big Tech]]> https://www.lambrospetrou.com/articles/big-tech-software-interviews/ big-tech-software-interviews Sun, 10 Sep 2023 00:00:00 GMT Over the past decade, I have gone through tens of interviews myself as an interviewee at small companies, startups, and Big Tech companies, from junior to Principal level roles. I also completed more than a hundred interviews as an interviewer during my time at Amazon/AWS, Meta, and Datadog.

Recently, I had discussions with several friends and colleagues about interviewing for software engineering roles, from entry level to senior levels. Some of them were going through interviews right now, and others asking about how to get better in interviewing in general.

So, I decided that instead of copy pasting resources every time, and having the same discussions repeatedly, I should write an article putting down all the generic advice I would give them. This way, our 1:1 discussions could focus on the specifics of their role, company, and skill-set instead.

This is NOT an exchaustive reference of interview preparation material. This is the advice I give to friends and colleagues, and how I personally prepare for interviews.

It works for me, it worked for some of my friends, so it might work for you too.

Table of contents

Prefer a personalized 1:1 session for tips, or a mock interview?

Book interview preparation session 1:1 mock interview (coding or system design)

Proper preparation is worth it

There are myriad of angry engineers online crying out loud that the current state of software engineering interviews is a mess. Some excuses they usually use against the coding interviews (which I do not agree with - I am pro-interviews!):

  • Why would you ever put a tenured engineer, writing code for years, sometimes decades, through a 40-minute process writing code in a collaboration doc or a whiteboard…
  • Why would you ask interviewees to explain and describe systems that they would probably never built in their day to day job…
  • Why would you not just believe that if someone says they are ninja coders, they can actually write code…
  • Why would someone even spend 1-2 months preparing for a stupid interview just to get into a big tech company, if they never use any of that knowledge day to day…

I am not going to debate the pros and cons of the interview process in this article, although I should write one in the future.

🙏🏼 For now, get over it! Big Tech companies, and even smaller ones, follow a pretty much standardised hiring process.

And that’s a good thing! As long as you can devote some time to prepare.

Preparing for the interviews, and actually getting a job in a big tech company often means 3-10x times higher total compensation versus a local small company 💰💶

For me, and I am sure most people out there, this is effing important.

Do you not care for money, or getting a job in Big Tech, fine, stop reading now, close the tab, and/or send me an angry tweet how much you disagree.

Note: Big Tech means FAANG in some circles, but in this article I refer to most multi-billion-dollar-revenue tech companies (e.g. Datadog, Cloudflare, Meta, Google, Stripe, Shopify).

Interview process

The interview process in these companies is (mostly) the same. There are small (or big) differences at the behavorial aspect of the interviews, and the difficulty of questions asked. Other than that, preparing for the top-tier companies will make your interviews with smaller companies a walk in the park.

Usually the whole interview pipeline consists of:

  1. Coding (2-3x)
  2. System Design - Architecture (1-2x) — for Senior+ levels
  3. Behavorial (1-2x)

I am going to focus on the above parts of the interview process. Each interview usually lasts 45 minutes, with some companies extending it to an hour.

Some companies might have interviews that are more conversational, or oriented around specific technologies (e.g. Java and JVM internals, machine learning algorithms and frameworks). Those need dedicated preparation and it really depends on the company you are interviewing with, so I am not going to get into those.

Coding

Coding interviews are the ones getting the most backlash by interviewees. Unless you are working in some deeply technical core Computer Science role, you are probably not exercising your coding problem solving skills at the level required for these interviews.

Therefore, you need to dedicate time preparing for this interview.

How much, depends solely on your skill, experience, and how often you practice.

As an example, after 5 years of working at Amazon/AWS, even though I worked at a technical product (lots of tree traversals, data transformations), I had to spend about 1 month practicing problem solving (almost) daily. This is because for 5 years, I had zero practice in coding interviews.

Next time I switched jobs was 2.5 years later, and that time I only needed about 2 weeks of preparation, mostly reading past material to refresh my memory. That’s because I had been doing 1-2 interviews every year just for practice, and was solving a few problems every couple months.

Coding - during the interview

During this interview, you will usually use a coding collaboration tool like Coderpad or HackerRank where both you, and your interviewer can collaborate on the code.

Initially, the interviewer will paste (or just verbally explain) the problem statement.

Once they give you the problem, this is where the “dance” starts, and it’s your turn to shine 😉

1. Ask clarification questions

As straightforward as it sounds, I interviewed many candidates that completely skipped this step and jumped into the code. Needless to say that most of those candidates failed spectacularly.

You have to ask clarification questions. It’s literally an evaluation checkbox in the interviewer’s handbook for your feedback.

Ask questions.

  • edge cases
  • size of the data
  • invalid inputs
  • input and output formats expected

Keep asking questions until the problem is 100% clear in your mind!

2. Explain your solution with examples before code

Another mistake candidates often do is not explaining their full solution before coding it.

There are several benefits in explaining your solution verbally, and actually testing it with at least one example input.

  • It’s important that you show how you think. How you approach a problem and walk through different solutions until you find the correct one.
  • If you missed something in your solution, explaining it might reveal it.
  • If you run out of time coding the solution, your interviewer will still know that you actually did think of a correct solution.
  • If you run out of time and cannot test your code at the end, you still need to show your interviewer that you can test an input with your solution and prove that it works. Do it early to get it out of the way.

This shouldn’t be a 10-minute discussion. It has to be a very short illustration, max 2-3 minutes, of how the solution works with an example input.

If you cannot find a solution within the first 2-3 minutes, it’s crucial that you keep talking.

Keep talking your thoughts, what do you have trouble with, which part of the solution works, and at which part are you stuck. You need to keep the interviewer engaged and in-sync with your thinking.

This not only gives you points in working-together and collaboration, but more importantly makes it trivial for the interviewer to steer you in the right direction in case your partial solution is completely the wrong one, or to give you the hint missing to fully solve the problem.

Don’t take the above point to the extreme either. If you need 1-2 minutes to think for yourself, that’s totally fine. Tell the interviewer that you want 2 minutes to think of it, or write some notes down. You just need to keep the interviewer in the loop.

Before moving on, I also ask the candidates to tell me the complexity of their approach in terms of time and space. After writing the code, I ask them again to reason about its complexity and compare it with the original answer.

3. Write the code

At this point you should have a clear idea of the solution you want to code. You should never start coding before you have a crystal clear understanding of the problem, and its potential solution.

Throughout the session, remember that you need to keep the interviewer in-sync with what you are thinking and doing.

While you write the code, say the highlights of what you are doing. For example, if you are going to iterate a list and do some transformation, say that and then write the code for it.

Many candidates have trouble knowing what to say and when while coding. One easy way for me to think about this, is that if I would write a comment in the code normally, that’s what I say out loud.

Random tips:

  • In these 10-15 minutes you usually just write 10-30 lines of code. You don’t need fancy classes, and hierarchies of inheritance. However, you do need to write simple and clean code that is easy to understand.
  • Write small functions to abstract away complexity from the function that solves the main problem. For example, if you need to iterate a string and parse something as part of a bigger solution, move the iteration and parsing in its own function to keep the main business logic simple.
  • If you forget about a specific language method name, or what the arguments are, don’t panic. Tell the interviewer something like “I know that there exists a method YYY, but cannot remember its definition. Can I write XXX to represent that method and come back to it later?”. Almost always the answer will be yes. If the tool supports running and evaluating the code, then this problem goes away since you can try and find the right method. Also, some interviewers will allow you to even google the method name (make sure to ask if you can do that though!).
  • If you are able to run the code written, then put print/console.log statements in your code and run it judiciously to make sure it works.
  • Don’t spend more than 2-3 minutes at the same line of code, typing and deleting without progress. Stop and think if needed. Ask the interviewer a question. Try to get out of your blackout.

Typing the code of the solution is usually the easiest part. You understand the problem. You thought of the solution. You just translate words into code at this point.

Remember to be friendly, engaging, and talk to the interviewer. If you do this, they will consiously, and unconsiously, help you. Either giving you a hint when they see you stuck, or just being there acting as your rubber duck.

4. Walk through the code and test it

Hopefully, you still have time after writing the code.

It’s important that you now do a quick walk through the code written, verbally explaining what each step does, and making sure it implements the previously discussed solution.

If you want to get all your points, you should also do a run-through with an example input. Show how the input will be processed at each line, similar to how a debugger in an IDE would work in a step-by-step execution.

Before moving on to the next problem, assuming the tool supports it, make sure to run your code and confirm that it does the right thing.

This can be done trivially by just calling your function with sample inputs and printing out the returned values. No need for fancy test frameworks to remember. For example, in Javascript this is how I do it:

function solve_problem_xxx(input) {
    // ...
}

console.log(solve_problem_xxx(/* input 1 */))
console.log(solve_problem_xxx(/* input 2 */))
console.log(solve_problem_xxx(/* input 3 */))

5. The interviewer interrupts you to move on

There are cases where the interviewer will interrupt you after 15-20 minutes to move on to a different problem.

This usually happens when they have more questions to ask you, and in the interest of time they want to move on to cover more topics. I usually tell the candidates how many questions we will do right at the beginning, so if your interviewer doesn’t say anything, it might be good to ask them yourself so that you can plan your time.

If you did all the above steps, you might not lose any points even if the code is not fully finished. You clarified the problem, explained the solution, reasoned about its complexity, showcased that you can do a dryrun, and you proved that you can write simple and clean code.

Having said that, if you only wrote very few lines of code, you will lose points.

This is a coding and problem solving skills interview, so you need to prove that you can solve problems, and write the code for their solution.

So, move with urgency, be fast, and be methodical. Follow the above steps, and practice.

Coding - preparation before the interview

There are myriads of coding interview books, like Elements of Programming Interviews in Python and Cracking the coding interview. There are also hundreds of online coding platforms dedicated to coding interviews like HackerRank, Leetcode (probably most famous).

I never liked Leetcode, and only use HackerRank from time to time to get some practice using an online evaluation tool instead of coding locally on my laptop.

After reading tons of books and trying these platforms, I personally wholeheartedly recommend to everyone that asks me how to prepare, to buy the Elements of Programming Interviews in Python book.

It has versions in other languages (e.g. C++, Java), but I always recommend the Python version since the answers are very simple Python code that most programmers should understand and be able to translate in the language they use.

This book contains a lot of problems for every category of questions used in the interviews. The problem difficutly ranges from easy to super hard.

There is even a sample guide at the front pages that suggests problems to solve from each category depending on how much time you can spend on preparation. This is super useful to give you an idea of which problems are really core, and which ones can be left for later.

What I like the most about this book, is that there is an explanation of the solution, and the solutions are often small and clean. This is in stark contrast to the “Cracking the coding inteview” book that used to have tens of lines of Java classes that just take away focus from the actual problem being solved.

Disclosure: I am not affiliated with the authors of this book, nor do I get any commission promoting it. I honestly just love it, and use it as my sole preparation material for coding interviews for the past 4 years, with success.

Need to practice

No matter the book, or the platform, or any other resource you use preparing for the coding interview, the only constant is that you need to practice.

Unfortunately, these problems are not something you do on a daily basis (in most roles), and therefore you need to spend time practicing in order to get good at them.

You need to solve a few problems of each category, so that later you can pattern match any given problem to something you previously did. Even if you don’t get identical questions in the interview, you will most likely use techniques that you came across while preparing, which makes a huge difference.

Put the time. Practice daily, or multiple times per week, for 2-5 weeks depending on your skills.

System Design

System design interviews are usually done only for the Senior level and beyond.

These are discussion oriented interviews where you are given a vague problem statement, and you need to come up with a design of a system architecture that solves that problem, and go deep into the technical details of the system.

For example, a common question is asking you to design YouTube.

Of course, you cannot just implement YouTube or come up with the absolute best architecture for it in 40-50 minutes. However, you need to show your skills in thinking about systems, limitations, constraints, and making reasonable assumptions.

During this interview you will use an online whiteboarding tool like Excalidraw, or if the interview is onsite a real-life whiteboard ✍🏼

There are different variations of this interview. There is the traditional backend/distributed systems interview, the mobile app design interview, the machine learning system design interview, and others.

Even though I am focusing mostly on the backend system design interview below, everything applies to the rest as well. The steps, the area of focus, and the tips apply to all the variations.

System Design - during the interview

Similarly to the coding interview, the system design interview can be tackled systematically, making it easier for you to prepare and handle the interview without depending too much on the interviewer steering of the conversation.

1. Explore the problem and ask questions

Once you get the problem statement, you have to spend the next 5-10 minutes asking questions.

You need to explore the problem as deep and as broad as possible. There are two kinds of questions you should do:

  1. Business requirements (also known as functional requirements)
  2. Technical requirements (also known as non-functional requirements)

Business requirements

This is where you will define the exact problem you will solve, what use-cases to support, what functionality to provide. Example questions (assuming the YouTube scenario):

  • Who is the user of the product?
    • viewers, video editors, ad publishers, …
  • How often and when do they use it?
    • 24/7 vs business hours, timezone based, global, …
  • What can they do with it?
    • upload/view/edit video, comments, download video, like/dislike, playlists, …

Technical requirements

This is where you will understand the scale and constraints of the system you should design. Example questions (assuming the YouTube scenario):

  • How many users per second?
  • How many videos “actioned” by each user?
  • Size limit per video?
  • Acceptable latency per operation?
  • Eventual consistency vs synchronous actions

Overall, after this series of questions you should know exactly what the product should do, and the constraints.

The interviewer might tell you to make assumptions instead of answering with a concrete value in some of your questions. In that case, try to give a reasonable guess based on products you know in real life (e.g. Facebook users around 2 billion).

The functional requirements are usually a much more limited set of the products you know and use, you would never propose a design for the whole of YouTube. But you can design a system for uploading, and viewing videos.

The non-functional requirements almost always revolve around the following dimensions:

  • Data size (ingestion, storage, processing)
  • Throughput (requests per second, number of users, read vs write ratio)
  • Latency and consistency (eventual vs synchronous consistency, asynchronous vs synchronous operations)
  • Cost (efficiency of the design)

It might be necessary, and almost always suggested, to do some back of the napkin math to estimate number of requests per second, storage needed, and other numbers throughout the session.

For latency-related estimations, use the handy comparison table in “Latency Numbers Every Programmer Should Know“, and during your calculations use rounded numbers to simplify.

2. Provide a high-level end-to-end design

Once you know what the system should do, for the following 5-10 minutes, the goal is to put some high-level design down.

This is important, and I have seen many candidates skip this step, and failing the interview in the end because they ran out of time without having an end-to-end system in-place due to spending too much time in a few components.

Here, you start talking about the main parts of the system. You are not going into technical details now. You describe the inputs of the system, main components of the system, and then the output of the system.

For example, for the YouTube scenario, assuming we only need an upload video and a view video page, this could be an initial high-level diagram.

High level architecture

You should not spend more than 10 minutes in this step.

The goal is not to cover every nitty gritty detail of the system, but to show that you understood the problem, you know the main components of the system, and the flow of data from input to output is clear.

3. Flesh out details for each component

This step should take roughly 1/3-1/2 of the interview duration, 20-25 minutes.

You now have to take each component of the high-level design and go one step deeper, fleshing out enough technical details, such that if someone took your diagram and notes after this step they would have a good idea how to start implementing your system.

I recommend you start from the input of the system, and walking through to the outputs, so that you stay focused. Do not jump from one component to another without being methodical, otherwise you will get confused and leave important things out.

For example, start by introducing load balancers in front of users.

Discuss the routing technique you use if there is any specific requirement. Do you need any stateful load balancing, e.g. sticky sessions, or is it purely stateless? Talk about these things as you draw.

Then, you move to the next component in our diagram above, the upload service.

How does the video uploading work? Probably you need be able to handle GBs of data being uploaded. Do you have a way to do it in parallel by splitting the video client-side, or is it all uploaded at once, or is it a multi-part upload?

Then, how does the video move to the transcoding service. Does the upload service store it temporarily somewhere else like Amazon S3, and only pass the object key to the transcording service? What’s the output of the transcoding service?

I hope it’s clear that in this section you go much deeper. You discuss many technical details as you progress through the system.

General tips

  • You should drive the interview. Don’t stop talking unless the interviewer interrupts you or asks you something. Show that you can control the interview and know how to describe a system. Do not just wait for them to ask questions and steer you towards a specific path.
  • Mention anything that comes to mind, but only draw and focus on main technical details and say that you will revisit the extra specific details in a follow-up round of deep-dives.
    • For example, you shouldn’t spend 10 minutes discussing load balancing algorithms if that’s not the main problem being solved.
  • You again need to cover the system end-to-end. The steps after this one will give you extra points, but this step is the meat of the interview.
  • Remember to justify your decisions as you go.
    • For example, if you say that you use Amazon S3 for storing the video, explain the properties it provides and why it suits your needs.
  • There are things that you probably won’t have expertise. You should still cover what you know, and explicitly mention what you don’t know.
    • For example, one time I told the interviewer I didn’t know about exact streaming algorithms for videos, but I know they exist, so I would use one of those. I explained that I knew that videos are delivered in chunks, and that there are some manifest lists for the chunks of a video and the player requests the right chunks, etc.

Important aspects to flesh out:

  • Load balancers and stateful/stateless scaling.
  • Databases used and why, e.g. NoSQL vs SQL RDBMS, data schemas.
  • Caching and database sharding.
  • Data flow from one component to the next one.
  • Point out explicitly if there are message queues (e.g. Amazon SQS), event streaming like Kafka, or synchronous gRPC calls.

4. Discuss about constraints - limitations - special considerations

At this point, the system should be well-defined with enough detail, and all the components are fleshed out.

For the next 5-10 minutes, you should start discussing about constraints of the system, its limitations, and single points of failure.

Examples:

  • If you use a cache what happens if it crashes?
  • Can the database selected handle the expected load?
  • How is the system impacted if each component fails, i.e. which parts are single points of failure?
  • What does recovery look like when a server crashes during video transcoding?
  • Discuss optimizations needed at scale, e.g. using a Content Delivery Network (CDN) to offload the delivery of the video parts from your servers.
  • What if we want to support 2x the load, or 10x the customers?

5. Deep dive into specific components

At this point, I usually have about 5 minutes remaining, and I ask the interviewer if they need me to go deeper in a specific component or if they have specific questions.

If they say no, don’t finish it here. Show your expertise in building systems, and pick a component and go deeper.

Focus on some of the aspects you brought up in the previous section and explain how you would tackle them.

If they say yes, then focus on that component and open up the discussion to them at this point, making it a dialogue.

Guidelines

The following table provides a summary of how you should approach the system design interview, based on the previous sections.

You can of course deviate depending on the company-specific details, but you should still apply the same structured thinking, methodical end-to-end designing, and deep dives into key components to showcase your technical depth.


Time spent Notes
5-10 minutes Explore the problem and ask questions. Focus on business requirements (functional) and technical requirements (non-functional).
5 minutes Provide a high-level end-to-end design. The flow of data and actions should be clear end-to-end without too many technical details.
25 minutes Flesh out all technical details for all the components. Start from the input of the system, all the way to the outputs. If someone took your diagram and notes after this step they would have a good idea how to start implementing your system.
5-10 minutes Discuss about constraints, limitations, crash recovery and fault-tolerance, special considerations depending on the problem.
5-10 minutes Deep dive into specific components if there is enough time. This section can be skipped if the previous one justifies taking more time.

Overall, keep in mind you will be evaluated for the following criteria:

  • Problem exploration and navigating ambiguity.
  • Understanding requirements and providing a high-level solution.
  • Showcase technical depth and broad knowledge. How well do you know the technologies you choose and how well do you justify using them.
  • Ability to communicate clearly when describing technical solutions.

System Design - preparation before the interview

Preparing for the system design interview doesn’t have a single approach. It’s not as simple as the coding interview, which boils down to practicing more questions.

Practicing system design questions helps, but if you don’t know much, then you won’t know what you don’t know 🤯

What I recommend, and what I personally do is the following:

  • Look into how your existing company implements a lot of their complex systems. You probably have access to all the internal implementation details, even the people working on them, so you can ask questions. Looking at real systems and how they are implemented is tremendously useful. I personally learnt a lot by researching how several AWS systems are implemented.
  • Read engineering blogs from well-known tech companies. This is very vague, I know, and I also have trouble following blogs outside a couple. The following is what I study religiously and then search around in more companies depending on topic.
    • The Amazon Builders’ Library: This is one of my favourite resources for learning about distributed systems. These are technical articles taken out from actual Amazon/AWS systems. I actually saw many of the techniques described in these articles in real-life during my time at AWS, and that’s why I love them. It’s not just marketing bullshit.
    • Meta engineering blog: Meta’s engineering blog is among my favourites. It spans things from AI, to developer tooling, to core infrastructure platforms, to web-scale metric systems. Some of them are high-level and not very technical, but some are super nice.
    • Cloudflare engineering blog: Cloudflare writes amazing technical blog posts about lots of their infrastructure and products. They range from super deeply technical network solutions to high-level architecture designs.
    • Other company blogs: Datadog, Stripe, pick your favourite tech company.
  • Watch the Systems Architecture Interview videos by Jackson Gabbard.
  • Understanding Distributed Systems, Second Edition: What every developer should know about large distributed applications
    • I love this book. It does NOT go deep into the topics discussed, but it gives you a very broad coverage of many aspects around distributed systems.
    • This can be a great starting book that will expose you to the many topics you should be aware when designing systems, and then get other resources to go deeper in the topics you feel you have a gap.
  • Watch the Distributed Systems lecture series by Martin Kleppmann.
    • Amazing playlist by the author of “Designing Data-Intensive Applications” (DDIA).
    • This series covers core distributed systems concepts (e.g. logical clocks, consensus, replication, quorums) with crystal clear explanations.
  • Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
    • At this point, this is the bible of distributed systems.
    • This book is not for beginners, or for spending a few days to quickly go over topics. This is an in-depth technical book, focused on the data aspects of applications and database concepts in general.
    • I recommend you leave this last, unless it’s really matching the role you are interviewing for. But, you should definitely read it if you have the time.
  • The System Design Primer
    • Has a lot of information, examples of questions, and links to lots of other content to help with the system design interview.
  • After you have read the above (or even during), it’s time to practice more. Just pick any product you use on a daily basis, pick a specific subset of its functionality, and start brainstorming how you would design it.
    • Doing this a few times will help you develop intuition in common solutions and techniques as most system designs have similar components.
    • Try to first think a solution on your own before googling to find out information about the actual implementation.

Other resources

The above material should be more than enough to prepare for the System Design interview, but if you want more material, the following are some resources I used a bit (definitely not exchaustively).

Behavorial

This interview varies a lot company to company.

Some companies focus on project work, digging into specific work you did, extracting information about your contribution, project complexity, etc.

Other companies focus on the people aspect of things, extracting information about your decision making, conflict resolution, and collaboration skills.

Most companies focus on both 😅

Behavorial - during the interview

The behavorial interviews are not as straight-cut as the technical interviews explored above. Therefore, I will just focus on things that apply in general, and things you should be prepared for anyway.

  • Be honest. Do not lie that you did things you didn’t do. If they actually use your provided references, they might reveal your lies. In most cases though, if the interviewer is experienced, they will pick up your lying and will ask follow-up questions that either you will have to lie even more, or you won’t know the answer to. In either case, you lose.
  • Be clear and use simple language. Big companies have employees from countries all around the world, and even though English is probably the language used daily, most people have accents. Speak in simple terms so that you are always understood, and keep an ear out if the interviewer asks you the same thing multiple times throughout the interview, since it might be an indication that communication is not clear.
  • Don’t be an arrogant jerk. Some folks think they are gods of engineering. Even if you are, don’t put it in your interviewer’s face and show-off. Showcase your skills with concrete data, examples, deep technical explanations, without insulting your interviewer.
  • Be specific in your contributions. There are very few products or projects that are delivered end-to-end by a single engineer. When describing some work you did, make sure to explain the overall situation with the product/project, the team, but emphasize on what you did as well. As an interviewer I have to know what your contribution is to the project. I don’t care about the project you worked on per se, but about the work you did, or you didn’t.
  • Use the STAR method when talking about past projects. Explain the Situation of the project, the Task to complete, the Actions you took, and finally the Result.
  • If you get a question that you have no idea what to answer, say it. Don’t stay there hanging or saying something completely irrelevant. Help yourself by letting the interviewer know. If they insist, then try to ask questions to get more specific, until it’s something you can answer.

Overall, these interviews have the following goals:

  • Did the candidate contribute in a project in a significant way, and knows how to quantify that, and describe their work to someone?
  • Is the candidate someone that can collaborate in a team, and be a good colleague to the rest of the company?

Be honest, be clear and specific, and showcase your skills without arrogance.

Behavorial - preparation before the interview

  • Go down memory lane and find at least two projects you are proud of. Be able to answer any question around them. You should be able to use the STAR method to describe the project, what problem it solved and what was your contribution. You should be able to answer technical questions about the project too, so spend some time reminding yourself about specifics of the project.
  • Prepare answers for the most common behavorial interview questions. Some of these questions are horrible, I hate them myself too, but many companies ask them. So, prepare before hand. These are the known “Tell me about a time” questions (1, 2). Some examples:
    • Tell me about a time you had a conflict with a colleague and how you resolved it.
    • Tell me about a time you had to solve a complex problem.
    • Tell me about a time your actions led to a negative outcome, and how did you recover.
    • Tell me what your colleagues would say as your best quality.
    • Which soft or hard skills would you like to improve on.
  • Study the company you are interviewing with, and try to sell yourself in a way that makes sense for them. For example, if the company is an analytics company, try to talk about a project or something you did and how it impacted the analytics of your product, or talk about work you did to improve the analytics you gathered to improve decision making.
  • Watch this Intro to Behavioural Interviews video by Jackson Gabbard.

Conclusion

I hope the above information helps even a tiny bit your preparation for the interviews.

This is my approach, and how I prepare for interviews. It has worked well for me so far, so I am confident you can get something valuable out of it.

Good luck 💪🏼 and let me know if you found this useful or if it helped you get that job!

Prefer a personalized 1:1 session for tips, or a mock interview?

Book interview preparation session 1:1 session — coding or system design interviews.
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Merge two directories recursively]]> https://www.lambrospetrou.com/articles/merge-two-subdirectories/ merge-two-subdirectories Sun, 13 Aug 2023 00:00:00 GMT Problem

I want to copy all the contents of a directory (or folder) named source-folder into another directory named destination-folder.

Let’s walk-through an example below.

Assume source-folder contains:

file1.txt
file2.txt
directory1/file3.txt
directory2/file4.txt

And destination-folder contains:

file5.txt
directory1/filexyz.txt
directory2/file4.txt

The final structure I want in destination-folder should be:

file1.txt
file2.txt
file5.txt
directory1/file3.txt
directory1/filexyz.txt
directory2/file4.txt
  • The directory2/file4.txt will be the original that was already in destination-folder, hence ignoring the one from the source-folder.

Solution

An easy solution is to use the rsync command line tool, available in most Linux/Unix systems.

rsync -av --ignore-existing source-folder/* destination-folder/

Without the /* it just copies the source folder itself into the destination! So, you would end up with source-folder under the destination-folder/.

Note: According to https://unix.stackexchange.com/a/149986 the wildcard is not needed, but the trailing slash is.

Detailed explanation

  • rsync: This is the command-line utility used for synchronizing files and directories between different locations, either on the same system or between different systems.
  • -av: Combines the following two options:
    • -a (or --archive): This option is used to achieve archive mode, which is essentially a combination of several other options like -r (recursive), -l (copy symlinks as symlinks), -p (preserve permissions), -t (preserve modification times), and more. It’s a convenient way to ensure that files are copied with their metadata and properties preserved.
    • -v (or --verbose): This option enables verbose mode, meaning that rsync will display detailed information about the files being transferred and the progress of the synchronization.
  • --ignore-existing: This option tells rsync to skip copying files that already exist in the destination folder. If a file with the same name exists in the destination, it won’t be overwritten or updated.
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Fast Feedback Loop and Delayed Gratification]]> https://www.lambrospetrou.com/articles/fast-feedback-loop-vs-delayed-gratification/ fast-feedback-loop-vs-delayed-gratification Mon, 29 May 2023 00:00:00 GMT For the past few months I have been working on some side-projects that took way longer than I expected, or initially planned.

This was partly due to my mistake of keep expanding their scope, but also because I underestimated the time investment needed. Shocking! Planning estimates gone wrong…

These delays sometimes demotivate me, or even trigger second-thoughts if I should even be doing these projects in the first place. Should I spend my time somewhere else instead?

My projects being delayed is not the purpose of this article though.

Delayed Gratification

I was always the kind of person that delays instant short-term gratification for a better outcome in the long-term.

In my younger days, I was studying hard to learn things well, get good grades to get into a good university years after, and building the skills to get a nice job.

In sports (volleyball), I trained for weeks, months, or years, in order to feel the excitement, satisfaction, and that amazing dopamine rush when we won matches or tournaments.

In other words, I am used to working hard now, for a better outcome later.

This is what delayed gratification means, and is explained nicely in James’ Clear article “40 Years of Stanford Research Found That People With This One Quality Are More Likely to Succeed”.

Fast Feedback Loop

A fast feedback loop is most often mentioned as part of Agile methodologies for project management/execution, modern developer tooling and platforms that promise faster iterations, and more. In general, it’s a popular pitch line for “something” increasing your productivity and output.

To me, fast feedback loop means that whenever I do something, anything, I am able to observe some kind of response back soon thereafter. Some feedback that indicates if what I did was good, bad, or if it had any effect at all.

When I am writing software code, I want to have a very fast feedback loop between the time I write the actual code to the moment when I know it’s functionally correct. Having good tooling that allows me to run unit tests as fast as possible (or a live-updating UI) the moment I save my code changes is absolutely crucial to my daily workflow.

I also want a tight feedback loop from when I merge my changes to the time I get feedback from customers actually using them. I want to have a continuous, fast, reliable way of shipping changes to production, and then also a way to get feedback on how it’s doing. That comes through metrics we define, complaints reported from customers, or even social media.

The Realization of the conflict

Let’s get back to the realization I had. 😅

Having a fast feedback loop seems to be in conflict with delayed gratification.

  • How can you build something that requires months or years, when you need a little amount feedback, a little gratification, all the time?
  • What can keep you going through hard times, when there is nothing to show for it yet?
  • How can you justify spending so much time on something, without getting something in return?

Even though I have embraced delayed gratification for most of my life, I now seem to need a fast feedback loop for most things I do. I can confidently argue that this is a side-effect of my day job as a software engineer, and it’s now influencing my attitude towards life too.

I realized though, that these two powerful behaviors are actually not in conflict with each other.

I have always been getting fast feedback, even when working towards very long-term big goals.

With midterm and final exams at school, you get feedback every few weeks. At sports, I play local tournaments every few weeks/months, and my teams always played in league competitions spanning months with matches at least once a month. In my day job, I get feedback during local development with good tooling. I get customer feedback through fast iteration on features and observability of the system behavior through metrics every time we release something.

The rookie mistake I did with my side-projects is that I had set these humongous goals at the beginning, without any small iterative milestones that would be completed every few weeks.

I was missing that dopamine boost.

I call this a “rookie” mistake because I already knew about this principle. Splitting big chunks of work into smaller pieces is essential for my day job. I read about agile and lean development from myriad books. And I still did the mistake…

One of my main projects is to create an advanced online course for Continuous Integration and Continuous Deployment (CI/CD) (elementsofcicd.com). Building the content for several months now, felt like writing a book, where you cannot get customer feedback until the whole book is finished, printed, then sold, and then getting your gratification.

I made a mistake. I should have taken small sections of the course and publishing them as articles. Maybe making a few smaller courses out of the sections that stand on their own, and releasing them even if the main course is not yet ready.

This would give me feedback, satisfaction, and the gratification that would keep me going to build the bigger, more complete, course I initially planned.

Life rule

This is now a rule for me, at work, and at life.

If I want to get things done, I need to adjust my environment, and my tasks, such that I maintain a fast feedback loop whilst at the same time delaying my gratification for the ultimate goal.

Finding a way to get continuous little-gratifications, in order to be able to get that delayed huge-gratification, is crucial.

“Most people optimize for the day ahead. A few people optimize for 1-2 years ahead. Almost nobody optimizes for 3-4 years ahead (or longer).

The person who is willing to delay gratification longer than most reduces competition and gains a decisive advantage.

Patience is power.”

— By James Clear in “3-2-1: The value of nature, controlling your attention, and designing your environment”

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[The perfect PaaS: Does it exist? Or impossible to build?]]> https://www.lambrospetrou.com/articles/the-perfect-paas-exists-or-impossible/ the-perfect-paas-exists-or-impossible Thu, 02 Feb 2023 00:00:00 GMT Warning: This article is a combination of deep multi-month research, hands-on experience over almost a decade, and ideas, all combined into a nice elaborated ranting. Proceed at your own peril!

Table of contents

Intro

Over the past 5 months, I have spent my entire free time outside working hours, researching, trying, and reading documentation about every cloud platform product I could get my hands on. From Platform-as-a-Service (PaaS), to self-hosted PaaS, to Backend-as-a-Service (BaaS), to traditional server management tools, to Infrastructure as Code (IaC) tools, to the newer and trendier Infrastructure from Code (IfC) tools, and everything in between.

I was still left disappointed with the existing offerings.

At some point, I even started being serious about building the platform I want myself. And then, once my own needs are satisfied, making it a product for others to use as well. Because, what’s a better business than one that solves your own problem, right? 😁

In a nutshell, I was (and still am) looking for a platform to use in order to deploy my stupid no-revenue nobody-uses toy projects, but also my more serious need-it-to-always-run might-bring-me-money projects.

I am a software engineer by trade, and over the years I have used a variety of platforms. During my time at Amazon, I started with our pre-AWS internal system (Apollo), but then (thank God) we migrated on AWS where I used and built several abstractions with raw AWS Cloudformation (and sprinkles of Jinja), then AWS SAM, and then AWS CDK. Similarly, at Meta we had our own internal deployment platform(s), and now at Datadog, we have yet another internal platform built on-top of Kubernetes (K8s).

Even within Amazon, every single team was building directly on-top of AWS, so you can imagine every team building their own mini platform and abstractions. Some were doing it really well, and some were doing it horribly bad… The funny part was that the final result of each team’s infrastructure and their CICD pipelines looked similar (at least the good ones). It was then, during my time working with AWS, that I started imagining what the ideal platform would look like, having seen the things that worked well, and the mistakes that nobody should ever repeat.

After all this time, I still haven’t found the platform that gives me everything I want. There are a few that come really close, but then fail me in some way.

This question haunts me

Does the perfect PaaS exist, or is it impossible to build?

There are tens of startups launched in this domain every year (joke tweet 1, joke tweet 2), but they either shut down after a few months, or they pivot into something entirely different, or they are just not good enough.

Is the perfect platform just a mirage? Why doesn’t it exist? Or does it, and I just haven’t found it yet?

Almost everyone I know (including myself) uses a combination of at least 2-3 platforms, integrating them with Infrastructure-as-Code templates, and then managing and running those templates in yet another Continuous Integration (CI) platform.

This is OK if you work on one project, but I don’t want to do it for every single project I work on.

Am I wasting my time on something that cannot be built? Is every person, and every team, really all that different to justify each building their own platform?

The front-end folks managed to converge in beautiful platforms, with Cloudflare Pages, Vercel, and Netlify, providing a very compelling package. I like them, and I use them. But sometimes they are too restricted and get super pricy when you want to lift those restrictions (read my serverless platforms overview back in 2022), which is when I fallback to a full-stack platform again.

I still believe that the platform I imagine can satisfy the needs of many people, and many teams.

What do I want from a platform?

I want the Heroku developer experience (DX) on-top of AWS. That’s it, that’s the buy pitch.

AWS is super powerful, reliable, with pay-as-you-go pricing. I love it. However, the developer experience using AWS directly is abysmal. I experienced AWS from the inside, and I know they won’t fix this. At least not any time soon.

Some folks joke that Heroku is the Heroku experience on-top of AWS. But I disagree. Heroku being built on-top of AWS is an implementation detail I don’t care about. I want the products AWS offers (or similar), the EC2s, the ECSs, the Lambdas, the DynamoDBs, and the S3s. But, I want them packaged in a Heroku experience.

Pricing

Pricing is a huge thing for any company. Pricing defines, and filters, the customers a company will attract. Do you want the Enterprises, charge thousands. Do you want the indie developers, give stuff for free. The pricing model is what will make a company profitable, or kill it.

Having said that, for me, as a customer, I like it more when pricing starts from zero ($0) when I don’t consume any resources, and increases as I use the platform more.

This pricing model has many names and variations: pay as you go, metered, usage-based, and nowadays a core component of any serverless product.

If you don’t use something you don’t pay, and the more you use something the more you pay.

I want to be able to host my toy projects that have literally 0 requests-per-second for cents, but I am happy to pay more for the projects I care about being available.

While looking through products, I tend to compare their compute pricing with AWS EC2, and anything that is within 2x of EC2 costs feels OK to me.

For comparison, Heroku costs 5-10x (or higher) more than AWS once we go past the entry-level dynos (what they call their instances): Performance M for $250/dyno/month vs EC2 m6g.medium for $33.58/instance/month.

Most new platforms also fail in the pricing dimension, and are way more expensive than what I would pay for my workload. Some platforms take a cut of the AWS bill for the resources they manage, which to me is nuts. The value I get from their platform doesn’t depend on the instance types I use (ten instances of t4g.nano vs c6g.2xlarge) so why should my bill. Others charge per seat ($99/seat is getting quite popular) in addition to actual compute charges, which again seems off for me.

Pricing is tricky, and I totally understand that for a company that pays $50K+ a year for an engineer, it doesn’t matter if the pay $1000 more for the tools they use. They need to make a profit at the end of the day. Personally, I prefer usage-based pricing models, even if the unit-price is a bit higher in low volumes.

I thought about pricing a lot. The model I would choose if I ever built it would be two-fold:

  • Provide usage-based pricing that starts from $0, but with a higher price-per-unit.
  • Provide per-seat or volume-discounted pricing for bigger companies.

With the above two-fold model, anyone can start using the platform to make sure it works for them, pay for their usage along the way, and once they settle to use it they could switch to the second pricing plan. Some prefer predictability, some prefer usage-based, so why not both.

Basecamp actually just updated their pricing to a two-fold model, just this month. As of this writing, they offer a per-user $15/month plan, but also a flat $299/month plan for unlimited use. That’s what ideal pricing looks to me! In this case it’s not stricly usage-based, but their low per-seat price is pretty close to the above model.

I love how easy it is to start using and ramping up with managed pay-as-you-go products like AWS DynamoDB, S3, and even non-AWS products like the new Momento Serverless Cache that charges $0.15/GB and that’s it. One price to think; it can’t get simpler than that.

Compute

This is the dimension where most of the platforms have the most restrictions, or limitations.

I am not going to focus on serverless functions in this article (although similar things apply), since a few things are more different when you go with a full-on serverless architecture (cold-starts, event-driven services, queues, frameworks, etc).

Coincidentally, exactly one year ago I wrote a deep dive article for AWS Elastic Beanstalk, explaining why I liked it, and elaborated on its feature set along with some of its nuances. That article was actually triggerred by this tweet I made voicing once again my frustration in finding my ideal platform (@LambrosPetrou/status/1487493396566528007).

There are three types of services that I want to run:

  1. The toy project that is fine to have a bit of cold-start, and doesn’t receive much traffic.
  2. An application that needs to be available, to have low latency (<100ms), and potentially auto-scale (1-5 nodes).
  3. An application (e.g. Gitea) that needs persistent disk access, hence a single-instance server, and which ideally shouldn’t have (long) downtime during deployments.

For the first type, I always use serverless functions with AWS Lambda, or Cloudflare Workers.

The second and third types is where restrictions come in play.

Compute / Requirements

  • Managed platform: OS updates, host patching, and optionally language runtime updates
  • Single server instance should be possible (because not everything is web-scale)
  • Persistent disk volumes (i.e. Amazon EBS)
  • Deployments in-place on same instance with only 1-2s of downtime (maximum)
    • Restriction due to the disk volume only being accessible from one host at a time
  • Variety of instance types, from the low-cost T3 to the powerful ARM-based Graviton2 C6g
  • Autoscaled cluster possible when needing more than one instance

Compute / What is missing?

My evaluation of a platform most often looks like this:

  1. I need persisted disk volumes (e.g. for SQLite), and many platforms automatically fail (e.g. Google Cloud Run).
  2. I prefer a managed platform taking care of OS updates, patches, etc. In most cases, this automatically means using containers. I couldn’t find any platform outside AWS Elastic Beanstalk that does this without containers. I could also consider the various server management platforms (e.g. Linode) but even if they offer patches and updates, it’s not as convenient as a truly managed runtime, and they lack everything else too.
  3. Most platforms that survive up to this step will fail the in-place deployments requirement. In-place deployments are a must when you have a persistent disk volume attached, otherwise you end up with long downtimes (stop using instance, detach volume, attach volume to new instance, start using new instance). AWS Elastic Beanstalk does this with only 1-2s downtime. Alternatively, I could use my own custom setup with AWS CodeDeploy & EC2.
  4. We only reach this step when considering the autoscaled cluster application, and probably we are using containers since not many platforms support executables natively. This is OK.

Summarising, when I have a Go application executable to deploy, my choices are:

  • Single-instance: Use AWS Elastic Beanstalk, or do my own thing with AWS CodeDeploy & EC2
  • Autoscaled cluster: Use AWS Elastic Beanstalk, AWS ECS, Fly.io, or other container platform.

Continuous Integration & Continuous Deployment

This is a huge topic (I am building a course about it), and products vary from a very basic Continuous Integration (CI) feature, to complex Continuous Deployment (CD) pipelines.

CICD / Requirements

  • Support for multiple AWS accounts (if the platform deploys in my own accounts)
  • Support for multiple regions, and multiple stages (staging, production, waves)
  • Support for custom build steps, and potentially per region when necessary
  • Support for different deployment targets (see the #Compute section above)
  • Support for automatic approval workflows (e.g. acceptance tests)
  • Support for manual approval between stages (because not everything can be automated)
  • Support for automatic rollback using some monitor/alert/metric/workflow
  • Track commits progressing across the pipeline
  • Logs and metrics per build/deployment/job

CICD / What is missing?

Everything? 😅 Jokes aside, I believe that working with the internal Amazon Pipelines tool for almost 5 years spoiled me.

There is not much public information on the tool, and no, AWS CodePipeline is not similar, it doesn’t even come close. However, the following two articles (personal favourites) from the Amazon Builders’ Library give a very detailed explanation for the core aspects of how deployments happen inside Amazon/AWS.

At a first glance, the list of requirements above might seem a lot. I would argue that all the above should be part of every CICD pipeline, and in my experience, all the teams end up implementing something close to that on their own after a few months of working on a project. So, why not have it built-in to the platform. Almost all good pipelines at Amazon “looked the same”, albeit with their own steps, approval workflows, and monitors to track when to roll back.

Most cloud platforms do not support such complex pipelines at all. Usually, they only provide single stage build/deploy workflows off a branch, and they offload the CICD needs to external dedicated products.

The majority of projects I see default in using Github Actions. It’s indeed a really nice product, but it has many limitations, and requires you to buy into the more expensive plans to enable some core features. I believe that a major part of its success is mostly due to the network effects of being free and readily available for anyone on Github.

Gitlab CI and Buildkite are among the best products I found, and I explored more than 20 of them. They are also the only ones supporting dynamically generated pipelines (in Gitlab, in Buildkite). This is very useful when you have a fixed initial pipeline, but then some configuration in your code, or the output of a step, determines the next steps in the pipeline. This is not a common feature, but it’s really useful for a project I am working on that automatically generates the infrastructure and deployment stages based on some configuration during the workflow runtime.

However, both have their own issues, high pricing for Gitlab, and managing the server-fleet for Buildkite. The pricing for CICD products also varies a lot, from per seat pricing, to usage-based pricing, from affordable, to crazy expensive.

Flexibility and extensibility

This is very generic and I don’t have a list of requirements. However, it’s necessary for a platform to offer extensibility hooks to customise the feature set provided.

On the one hand, platforms like AWS Elastic Beanstalk provide very powerful hooks to override almost everything, and on the other hand, platforms like Google App Engine standard environments are very restricted and opinionated for anything you do.

Some things I want to able to customize that not all platforms support:

  • Compute resources (CPU, RAM)
  • Deployment strategy (in-place, green/blue, canary)
  • Autoscaling strategy (request-based, utilisation-based)
  • Filesystem and storage options (disks, nfs)

In a future ideal world, we shouldn’t need to specify CPU/RAM requirements either. Just run the application, and as requests come in it will get the resources it needs, and the platform will be able to absorb and distribute the load beautifully.

Good citizen

Another generic, and even more vague aspect I am looking for when evaluating a platform is if the platform is a good citizen within the ecosystem.

For example, if the platform comes with its own CDN, great. But can I proxy my deployed application with my own CDN if needed? Some Anycast platforms have issues in some setups.

Can I plug my own monitoring and observability product (e.g. Datadog) easily, or am I stuck with its own monitoring stack, if there is any at all.

Are there any hooks I can run generic commands/scripts for adhoc jobs I need during deployments, like provisioning external resources through Terraform? This would be a solved problem assuming that there is some kind of CICD workflow, but it’s something to watch for.

As I see it, the platform should provide good enough defaults to start with, whilst giving me hooks to customize my experience.

Developer experience (DX or DevX)

Developer experience is essentially how a platform feels like when using it, after considering every aforementioned dimension.

Is the platform intrusive to my application code? For example, do I need to change my business logic to be able to run on that platform? Examples here include App Engine, and most serverless function providers in some degree.

Can I accomplish my day-to-day tasks without wasting time reading badly written documentation, deciphering cryptic error messages, or fighting with broken CLIs and websites? Is it easy to troubleshoot and investigate issues with the infrastructure if and when they happen?

What are the abstractions of the platform? Do I need to configure every single piece of infrastructure similar to using AWS directly (e.g. VPC, subnets, autoscaling groups, scaling policies, IAM roles, IAM policies), or does it provide useful high-level abstractions with sane defaults. How much effort do I need to put to get my code deployed?

Do I really care that the platform is built on-top of Kubernetes (K8s)? No, I do not!

As a positive example, Fly.io with its built-in private networking which acts as advanced service discovery is really great.

Preview environments per pull-request, or easy to create environments per engineer, are crucial to have for faster feedback loop during development. This trend was commodized by the frontend platforms in the past few years, so now more platforms are supporting them.

This is just the tip of the iceberg of the things that make up a nice developer experience.

I have faith that the newer serverless products, alongside the managed AI platforms, will drive us towards a better future for traditional backend stacks too.

One of the most promising platforms I found was Modal, even though I usually dislike Infrastructure from Code tools. Modal combines amazing DX, great usage-based pricing, and they seemelessly disappear in your code. Unfortunately, they only support Python right now, but I am hoping they expand to more languages soon.

My complaint to AWS

Amazon and AWS have many issues at the company level, I can talk for months about them, but somehow AWS offers some of the most reliable and highly-available infrastructure in the industry.

If we focus on core products like EC2, S3, Lambda, DynamoDB, Route53, SQS, and a few others, you can pretty much build anything for insane scale. If only they fixed the developer experience of the platform.

They provide all the little pieces of the puzzle, without a preview of the final puzzle!

AWS Elastic Beanstalk was my favourite deployment platform a few years ago. Unfortunately, AWS abandoned it, and it shows. It was late to get Spot instances support, late to get Arm Graviton instances support, and there was hardly any major improvement of the product since its upgrade to the Amazon Linux 2 platform (back in 2020).

It combined everything, single instance, containers, managed autoscaled clusters, good default monitoring dashboard, simple CICD, and an amazingly extensible platform.

My guess is that the original internal team gradually moved on to other projects, or companies, and then the whole product entered maintenance mode. It’s not the first time this would have happened…

AWS wasted millions of dollars in numerous attempts to bring some of that Heroku experience in AWS, by building new services, instead of fixing, improving, and promoting Elastic Beanstalk. Services like AWS CodeStar (2017), AWS CodeCatalyst (2022), and even the promising AWS App Runner (2021), are all attempts to provide a better PaaS natively by AWS.

They are all going to fail. It’s obvious, to me at least, that most attempts after Beanstalk are mostly CV and promotion oriented projects. Who wants to get credit for improving an existing service, when they can advertise on LinkedIn that they launched a whole new service, and parade with announcement blog posts. Right? A new director can lead the launch of a new service, and grow a team from 3-4 people to 30 just within a year. That’s how directors get promotions. Been there, saw that, left after that!

Sometimes, starting with a clean sheet is the right choice, but in the case of Elastic Beanstalk, I believe that pushing it to the side until it rots, was the wrong decision.

Oh well, what do I know, I am neither Jeff Bezos, nor Andy Jassy 🤐

Conclusion

Okay, my ranting is almost over.

I probably looked into more than 50 products (see list of products/tools/platforms), open source self hosted platforms (e.g. Dokku, CapRover), platforms that deploy into my own AWS accounts (e.g. Qovery, Flightcontrol, WithCoherence, Stacktape), managed platforms (e.g. Fly.io, Render, Railway), and even the legendary Heroku.

All of them fail in some of the above core requirements. If it’s not pricing, it’s features, and if it’s not features, it’s developer experience.

I currently settled on the following mix:

  • Cloudflare Pages for websites.
  • AWS Lambda for anything that can tolerate cold-starts and >100ms latencies.
  • Fly.io for my autoscaled applications that work without in-place deployments.
  • AWS Elastic Beanstalk or custom EC2 setup for my single-instance use-cases.
  • AWS SAM and Terraform for Infrastructure as Code.
  • Combination of AWS CodeBuild, Github Actions, and experimenting with Buildkite, for CICD.

A simple decision tree for the platform I use depending on my needs:

Platform decision tree

I basically have to setup the same things for every project I am working on, no matter how small or big. Infrastructure as Code templates, a CICD pipeline in a separate tool (when I need more than one stage) that needs access to the repo and my cloud account credentials, moving shared stuff into separate packages or copy-pasting them every time across repositories, etc.

Why not move all this boilerplate into a platform. Is it only my boilerplate?

The question remains

The Perfect PaaS: Does it exist? Can it be built? Or is it impossible?

The platform I envision. The platform I want.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Hell Yeah or No - what's worth doing | Highlights]]> https://www.lambrospetrou.com/articles/hell-yeah-or-no-highlights/ hell-yeah-or-no-highlights Sat, 12 Nov 2022 00:00:00 GMT This article contains my favourite highlights from the Hell Yeah or No book from Derek Sivers.

I was tweeting these highlights while reading the book, but I wanted to persist them on my blog as well. Check the original Twitter thread at @lambrospetrou/status/1585311232940445701.

I strongly recommend reading the whole book. It’s very short, with each chapter just being a couple of paragraphs, but immensely satisfying.

4. Actions, not words, reveal our real values

No matter what you tell the world or tell yourself, your actions reveal your real values. Your actions show you what you actually want.

6. Why are you doing?

It’s crucial to know why you’re doing what you’re doing. Most people don’t know. They just go with the flow.

Whatever you decide, you need to optimize for that goal, and be willing to let go of the others.

8. Imitate. We are imperfect mirrors.

Like a funhouse mirror that distorts what it reflects, your imitation will turn out much different from the original. Maybe even better.

So look around at those existing ideas in the world.

11. Character predicts your future

How you do anything is how you do everything. It all matters.

Your actions are completely under your control, and seem to be the best indicator of future success.

14. Small actions change your self-identity

Your actions show the world who you are. You won’t act differently until you think of yourself differently. So start by taking one small action that will change your self-identity.

15. If you’re not feeling “hell yeah!” then say no

Say no to almost everything. This starts to free your time and mind.

Then, when you find something you’re actually excited about, you’ll have the space in your life to give it your full attention. You’ll be able to take massive action, in a way that most people can’t, because you cleared away your clutter in advance. Saying no makes your yes more powerful.

Refuse almost everything. Do almost nothing. But the things you do, do them all the way.

This is really one the top highlights!

19. Tilting my mirror (motivation is delicate)

When you notice your motivation fading, you have to seek out the subtle cause. A simple tweak can make all the difference between achieving something or not.

24. There’s no speed limit

[…] “the standard pace is for chumps” — that the system is designed so anyone can keep up. If you’re more driven than most people, you can do way more than anyone expects.

34. Switch strategies

Early in your career, the best strategy is to say yes to everything. The more things you try, and the more people you meet, the better. Each one might lead to your lucky break.

Then when something is extra-rewarding, it’s time to switch strategies. Focus all of your energy on this one thing. Don’t be leisurely. Strike while it’s hot. Be a freak. Give it everything you’ve got.

35. Don’t be a donkey

The solution is to think long term. Do just one thing for a few years, then another for a few years, then another.

Most people overestimate what they can do in one year, and underestimate what they can do in ten years.

Think long term. Use the future. Don’t be short sighted. Don’t be a donkey.

50. Don’t start a business until people are asking you to

Most have an idea but no customers. For them I always say, “Don’t start a business until people are asking you to.”

Don’t announce anything. Don’t choose a name. Don’t make a website or an app. Don’t build a system. You need to be free to completely change or ditch your idea.

Then you get your first paying customer. Provide a one-on-one personal service. Then you get another paying customer. Prove a real demand.

Then, as late as possible, you officially start your business.

66. If you think you haven’t found your passion…

It’s dangerous to think in terms of “passion” and “purpose” because they sound like such huge overwhelming things.

[…] just notice what excites you and what scares you on a small moment-to-moment level.

If you keep thinking about doing something big, and you find that the idea both terrifies and intrigues you, it’s probably a worthy endeavor for you.

You grow by doing what excites you and what scares you.

67. Whatever scares you, go do it

Fear is just a form of excitement, and you know you should do what excites you.

Best of all, once you do something that scared you, you’re not scared of it anymore! As you go through life, doing everything that scares you, you fear less and less in the world.

Life is an ongoing process of choosing between safety (out of fear and need for defense) and risk (for the sake of progress and growth). Make the growth choice a dozen times a day.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[AWS Lambda + Upstash Redis + Go = 🚀❤️]]> https://www.lambrospetrou.com/articles/upstash-redis-aws-lambda/ upstash-redis-aws-lambda Sun, 16 Oct 2022 00:00:00 GMT This article was originally posted as a guest post on Upstash Redis blog. I am cross-posting it here for my records.

Intro

Serverless compute platforms are awesome, but without serverless databases they are too limited.

While I was building the platform for my upcoming course, The Elements of CI/CD, I wanted a serverless database since I decided to use AWS Lambda as my server for certain things. The requirements I had were:

  1. Pay as you go pricing. I don’t want to pay per hour or per node, but by usage (requests, storage, etc). It should be very cheap to start using it and as usage ramps up the cost would grow proportionally.
  2. Low latency. Nobody likes slow responses, so querying the database should be fast from inside AWS regions (e.g. eu-west-1).
  3. Great developer experience (DevX). Having a nice interface for the database without having to learn yet another niche DSL, or wasting hours fiddling with a website is preferred.

Upstash Redis satisfies all of the above requirements, and it does a great job at it.

  • Pay as you go?
    • ✅ Really affordable to start, and also at scale!
  • Low latency?
    • ✅ <1ms latency from querying from inside AWS Lambda!
  • Great DevX?
    • ✅ It’s standard Redis. So, yeap.

In this article we are going to see how to use Upstash Redis from inside AWS Lambda, ensure it is fast enough for our needs, whilst at the same time keeping our code maintainable in order to be able to test locally or deploy to a different platform if needed.

What are we implementing?

For simplicity, we are going to implement just 3 API endpoints:

  1. The GET|POST /login endpoint which accepts a userId as a query parameter in a GET, or inside a form value submitted with a POST request. This endpoint will generate a session ID, store it in Redis, and also set a cookie for subsequent visits. The GET just makes it easier to test.🙃
  2. The GET /lessons/completed endpoint requires logged in users (i.e. having the cookie with the session ID) and returns a JSON response with all the lessons the user completed and when.
  3. The POST /lessons/{lessonSlug}/mark-complete endpoint requires logged in users (i.e. having the cookie with the session ID) and marks the lesson denoted by lessonSlug as completed with the current time.

Note: In the code below there are a few things missing, thus this is not production-ready copy-pasteable code. For example, we should check that the given lessonSlug exists before updating it. The login endpoint should also accept a password and do proper salted/hashed verification before creating session IDs, etc.

1. Setup

  • The complete code detailed below also exists in my aws-playground repository if you want to see how everything fits together.

As you will see below, we are creating two entrypoints, i.e. two executable commands. One will be for a normal local server, and one will be for AWS Lambda. This way, we will be able to test our whole logic locally, and with standard unit/integration tests if we wanted.

The only differences between them is shown in sections 1.2 and 1.3 below.

1.1 Workspace

Before we dive into the code, let’s setup our working directory for Go.

  1. Download Go.
  2. Create an Upstash Redis account and database in a region of your choosing. Ideally it should be the same AWS region you will deploy your Lambda. I will be using eu-west-1 (Europe, Ireland) in this article.

Redis database details

After completing the above, we can now create our workspace. For the rest of the article, assume our code is under ~/dev/aws-lambda-upstash-redis.

mkdir -p ~/dev/aws-lambda-upstash-redis
cd ~/dev/aws-lambda-upstash-redis

Then, create a Go package:

go mod init com.upstash/example/aws-lambda-upstash-redis

1.2 Local server entrypoint

  • Paste the following code in ~/dev/aws-lambda-upstash-redis/cmd/server/main.go.
package main

import (
    "log"
    "net/http"
    "os"

    "com.upstash/example/aws-lambda-upstash-redis/core"
)

func main() {
    mux := core.NewMux()
    port := os.Getenv("PORT")
    if len(port) == 0 {
        port = "5000"
    }
    if err := http.ListenAndServe(":"+port, mux); err != nil {
        log.Fatal(err)
    }
}

1.3 AWS Lambda entrypoint

  • Paste the following code in ~/dev/aws-lambda-upstash-redis/cmd/lambda/main.go.
package main

import (
    "com.upstash/example/aws-lambda-upstash-redis/core"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/awslabs/aws-lambda-go-api-proxy/httpadapter"
)

func main() {
    mux := core.NewMux()
    lambda.Start(httpadapter.NewV2(mux).ProxyWithContext)
}

1.4 Core logic

Our main core logic will go into the core package to be shared by both entry points above.

  • Paste the following code in ~/dev/aws-lambda-upstash-redis/core/lib.go.
package core

import (
    "github.com/go-chi/chi/v5"
)

func NewMux() *chi.Mux {
    r := chi.NewRouter()
    return r
}

1.5 Building / Compiling

I usually write a small makefile to avoid typing long commands every time I want to compile so copy the following into ~/dev/aws-lambda-upstash-redis/makefile:

default: build

clean:
    rm -rf build/

build: build-lambda build-server

build-lambda: clean
    GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o build/handler cmd/lambda/main.go
    cd build/ && zip handler.zip ./handler

build-server: clean
    CGO_ENABLED=0 go build -o build/server cmd/server/main.go

Don’t worry too much for now about the details but this allows us to do:

  • make build-server: Builds the binary for running the server locally (executable ./build/server).
  • make build-lambda: Builds the binary for running the server on AWS Lambda (executable ./build/handler and ./build/handler.zip).
  • make or make build does both.

The CGO_ENABLED=0 option makes sure our executable binaries are self-contained (i.e. statically compiled). The GOOS=linux GOARCH=amd64 options are needed to cross-compile and match the linux environment of AWS Lambda in case you are using a Mac or Windows system locally.

Next, run go mod tidy to fetch all the code dependencies. Remember to run this every time you add or remove Go dependencies.

Finally, run make once, in order to build everything and make sure your workspace is setup, before we go deeper into the code.

2. API implementation

For this section we will always be working inside the ~/dev/aws-lambda-upstash-redis/core/lib.go file.

The following few lines define the API endpoints we discussed earlier, using the amazing go-chi library.

import (
    //...
    "github.com/go-chi/chi/v5"
    "github.com/go-chi/chi/v5/middleware"
)

func NewMux() *chi.Mux {
    r := chi.NewRouter()

    r.Use(middleware.RequestID)
    r.Use(middleware.Logger)
    r.Use(middleware.Recoverer)

    r.Get("/login", login)
    r.Post("/login", login)
    r.Group(func(r chi.Router) {
        r.Use(UsersWithSessionOnly)
        r.Get("/lessons/completed", listLessonsCompleted)
        r.Post("/lessons/{lessonSlug}/mark-complete", markLessonComplete)
    })

    return r
}

In the snippet above, r.Group(...) creates a shared layer where we can apply common middleware for any route defined inside of it. In this case, we add our own middleware UsersWithSessionOnly, which as we will see later guarantees that the request contains the cookie with an active session ID.

2.1 UsersWithSessionOnly middleware

In this middleware we want to implement the following:

  1. Extract the cookie that contains the session ID, and fail otherwise.
  2. Query Redis to fetch the user details based on the session ID, and fail if the session ID provided is not active.
  3. Store the user ID in the request’s context.Context in order to make it available to downstream middleware or handlers.

First, we need some boilerplate code for some imports and definitions that are used everywhere.

import (
    //...
    "log"
    "os"
    "strings"

    "github.com/go-redis/redis/v8"
)

type contextKey struct {
    name string
}
const (
    COOKIE_AUTH_NAME = "xxx_session_id"
)
var (
    CTX_USER_ID = &contextKey{"LoggedInUserId"}
    redisDb     = NewClient()
)

func NewClient() *redis.Client {
    redisUrl := strings.TrimSpace(os.Getenv("UPSTASH_REDIS_URL"))
    if redisUrl == "" {
        log.Fatalln("Required env UPSTASH_REDIS_URL not set!")
    }
    opt, _ := redis.ParseURL(redisUrl)
    redisDb := redis.NewClient(opt)

    return redisDb
}

And now the main logic for the authentication middleware.

// UsersWithSessionOnly middleware restricts access to just logged-in users.
// If validation passes, then the context will contain the user id (CTX_USER_ID).
func UsersWithSessionOnly(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        c, err := r.Cookie(COOKIE_AUTH_NAME)
        if err != nil {
            render.Status(r, http.StatusForbidden)
            render.JSON(w, r, struct{}{})
            return
        }

        ctx := r.Context()
        userId, err := redisDb.Get(ctx, "session:"+c.Value).Result()
        if err == redis.Nil {
            // If session is not found then user is forbidden from accessing the API!
            render.Status(r, http.StatusForbidden)
            render.JSON(w, r, struct{}{})
            return
        } else if err != nil {
            // Something went wrong querying Redis!
            render.Status(r, http.StatusInternalServerError)
            render.JSON(w, r, struct{ Message string }{Message: "We could not validate the provided session ID"})
            return
        }
        // Set it for downstream middleware and handlers.
        next.ServeHTTP(w, r.WithContext(context.WithValue(ctx, CTX_USER_ID, userId)))
    })
}

2.2 markLessonComplete(…)

This is a straightforward operation where we want to store in Redis that the lesson denoted by the lessonSlug path parameter is completed at the current time of the request.

In Redis we want to keep a map for each user where each key-value pair in the map will be the lesson as key, and the completion date as value. Therefore, we use the HSET Redis command. We could also store a separate key per lesson but this makes it easier to fetch all the lessons for a user at once later.

func markLessonComplete(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    lessonSlug := chi.URLParam(r, "lessonSlug")
    userId := r.Context().Value(CTX_USER_ID).(string)
    timeNow := time.Now().Format(time.RFC3339)

    err := redisDb.HSet(ctx, "lessons:"+userId, lessonSlug, timeNow).Err()
    if err != nil {
        render.Status(r, http.StatusInternalServerError)
        render.JSON(w, r, struct{ Message string }{Message: "We could not save your progression..."})
        return
    }

    render.JSON(w, r, struct {
        LessonSlug    string
        LastCompleted string
    }{
        lessonSlug,
        timeNow,
    })
}

2.3 listLessonsCompleted(…)

In similar fashion as the previous section, here we just want to return the whole map of lessons completion and return it to the user in a JSON response. We use the HGETALL command for this.

func listLessonsCompleted(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    userId := r.Context().Value(CTX_USER_ID).(string)

    lessons, err := redisDb.HGetAll(ctx, "lessons:"+userId).Result()
    if err == redis.Nil {
        lessons = map[string]string{}
    } else if err != nil {
        render.Status(r, http.StatusInternalServerError)
        render.JSON(w, r, struct{ Message string }{Message: "We could not load your lessons..."})
        return
    }

    render.JSON(w, r, struct {
        Lessons map[string]string
    }{
        lessons,
    })
}

2.4 login(…)

Finally, the login endpoint. Once again, please do not copy the following code into production, since it’s not doing any kind of validation. For the purposes of this article we are only interested in how it queries Redis and how it sets the cookie for the session ID.

The session ID is generated by the ksuid library, which has a few advantages over normal UUIDs, and we only consider it active for 1 hour. We use the time-to-live functionality of the Redis SET command for automatic removal from the database after one hour.

import (
    // ...
  "github.com/segmentio/ksuid"
)

func login(w http.ResponseWriter, r *http.Request) {
    // Check credentials and update redis session and return Set-Cookie
    // WARNING: You should do an actual validation in production for credentials!
    // ...
    // For now we always assume correctness and automatically create a session token
    // by saving it to Redis, and also setting it as a cookie.
    userId := strings.TrimSpace(r.FormValue("userId"))
    if userId == "" {
        render.Status(r, http.StatusBadRequest)
        render.JSON(w, r, struct{ Message string }{Message: "Missing required userId"})
        return
    }

    sessionId := ksuid.New()
    redisDb.Set(r.Context(), "session:"+sessionId.String(), userId, time.Hour*1)
    http.SetCookie(w, &http.Cookie{
        Name: COOKIE_AUTH_NAME, Value: sessionId.String(),
        Path: "/", MaxAge: int((time.Hour * 1).Seconds()),
        // This should be true when deploying in production (https), but locally we need it false (http).
        Secure: false,
    })

    http.Redirect(w, r, "/lessons/completed", http.StatusTemporaryRedirect)
}

3. Demo - Locally

Phew, that was a lot of code.😅

Let’s do a quick demo to make sure everything works as expected.

  • First, set the UPSTASH_REDIS_URL environment variable to the URL of the database you created in section 1 above. You can find it in the details tab of your database’s page (see section 1.1 above).
export UPSTASH_REDIS_URL="<your-url-here>"
  • Then, build and run the local server:
make build-server && ./build/server

Browser testing

Now let’s do some testing in the browser by visiting http://localhost:5000/lessons/completed.

Demo access denied

We get 403 - Forbidden, so let’s login, by visiting http://localhost:5000/login?userId=lambros.

Demo login results

We are logged in now, and we automatically got redirected to /lessons/completed, but they are empty. So, let’s mark a lesson as completed. In the console tab inside the devtools of your browser run the following:

await (
  await fetch("http://localhost:5000/lessons/123/mark-complete", {
    method: "POST",
    credentials: "same-origin",
  })
).json();

// Should output something like:
// {LessonSlug: '123', LastCompleted: '2022-10-12T02:01:14+03:00'}

Visiting http://localhost:5000/lessons/completed should show this lesson as marked now:

{ "Lessons": { "123": "2022-10-12T02:01:14+03:00" } }

Et voila. Everything works fine!

Looking into the Redis database itself using the recently launched online Data Browser also proves that the expected data is there.

Redis Data Browser

4. AWS Lambda

In order to test and deploy to AWS Lambda we are going to use the sam cli.

  • First, setup the SAM cli and make sure your user/role has the right permissions.

  • The sam cli needs a Cloudformation template to work, so copy the following into aws-iac/sam-template.yml:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Defines all the AWS resources we need for our Upstash Redis API.

Resources:
  # https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html
  GoUpstashRedis:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../build/handler.zip
      Handler: handler
      Runtime: go1.x
      MemorySize: 512
      FunctionUrlConfig:
        AuthType: NONE
        Cors:
          AllowCredentials: false
          AllowMethods: ["*"]
          AllowOrigins: ["*"]

Outputs:
  GoUpstashRedisApi:
    Description: "Endpoint URL"
    Value: !GetAtt GoUpstashRedisUrl.FunctionUrl
  GoUpstashRedis:
    Description: "Lambda Function ARN"
    Value: !GetAtt GoUpstashRedis.Arn
  GoUpstashRedisIamRole:
    Description: "Implicit IAM Role created for GoUpstashRedis"
    Value: !GetAtt GoUpstashRedisRole.Arn
  • Build the handler bundle for AWS Lambda:
make build-lambda
  • Add the following to makefile to make it easy to deploy after we do code changes:
sam-deploy: build-lambda
    sam deploy -t aws-iac/sam-template.yml --stack-name "UpstashRedisGoArticleStackDemo" --region eu-west-1 --resolve-s3 --no-confirm-changeset --no-fail-on-empty-changeset --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
  • Deploy to the specified region (see previous command).
make sam-deploy
  • You should get some output like the below:
CloudFormation outputs from deployed stack
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 GoUpstashRedis
Description         Lambda Function ARN
Value               arn:aws:lambda:eu-west-1:<redacted>:function:UpstashRedisGoArticleStackDem-GoUpstashRedis-baB8dQPkTfg0

Key                 GoUpstashRedisIamRole
Description         Implicit IAM Role created for GoUpstashRedis
Value               arn:aws:iam::<redacted>:role/UpstashRedisGoArticleStac-GoUpstashRedisRole-16UWC7HR6KII8

Key                 GoUpstashRedisApi
Description         Endpoint URL
Value               https://6pmmwqmg5vec3bcsldabckaf5i0nlgje.lambda-url.eu-west-1.on.aws/
-----------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - UpstashRedisGoArticleStackDemo in eu-west-1
  • The URL of the deployed AWS Lambda is shown in the output printed, in this case https://6pmmwqmg5vec3bcsldabckaf5i0nlgje.lambda-url.eu-west-1.on.aws/. So, feel free to repeat the demo steps we did in our browser before using localhost with the actual domain now.
    • Alternatively, you can also find the URL of the newly created function in the outputs of the CloudFormation stack UpstashRedisGoArticleStackDemo in the Cloudformation console.
    • Note: Make sure to set the UPSTASH_REDIS_URL environment variable on your AWS Lambda configuration as well, otherwise it will just crash. Visit the AWS Lambda console, then click on your newly deployed Lambda, click on the Configuration tab, and then on the left side menu click Environment variables. Type UPSTASH_REDIS_URL as key, and your Upstash Redis URL as value. Click Save, and now your Lambda is ready.

4.1 SAM local test

We can test our Lambda locally by providing a sample-event.json with the right path/cookie/query parameters/etc. Example of such JSON can be found in aws-lambda-upstash-redis-article/sample-event.json.

  • Then, once you have a valid JSON event file, run the following to invoke the server logic as it would run on AWS Lambda:
sam local invoke -t aws-iac/sam-template.yml -e sample-event.json

4.2 Security of Upstash Redis URL

In this article, for simplicity we provided the Upstash Redis URL, containing the password, through environment variables. We don’t want to hardcode this into the SAM Cloudformation template which is versioned along with our code, hence why we had to manually configure it through the AWS Lambda console.

There is a better way to do this automatically without modifying the Lambda configuration every time and to avoid having the Redis credentials/URL in plain sight for anyone with console access.

We can use AWS Systems Manager Parameter Store and the corresponding Cloudformation resource AWS::SSM::Parameter to hold the URL (we can set it once and retain during deployments), and change our Lambda code to fetch the parameter’s value at runtime. We could also automatically inject it as an env variable inside sam-template.yml, although this would still have it in plain text in the console.

Changing the code to fetch it from SSM Parameter Store is easy due to our separation of entrypoints, so we could fetch the parameter only when running inside AWS Lambda (~/dev/aws-lambda-upstash-redis/cmd/lambda/main.go) and passing it through to the NewClient() function that creates the Redis client.

5. How fast is it?

Apart from the first cold start invocation which takes roughly 100-120 ms, every invocation thereafter is lightning fast and always under 4 ms.

Below is an example of a hot invocation for the /login?userId=lambros endpoint as implemented above:

AWS Lambda runtime duration

As you can see the total duration of our request was 2.06 ms. Yes, that’s two milliseconds, to generate a session ID, write it to Upstash Redis remotely, and return the redirection response.

Looking more carefully in the Log output section, we can see that the request lasted 789.435μs, at least from our code’s perspective. This means our logic completed well under 1 ms (roughly 0.790 ms). Mind-blowing considering we are using a remote database.🤯

Conclusion

I am really amazed by how well Upstash Redis performs, especially since it’s hard to find such performance for serverless databases suited for platforms like AWS Lambda.

The Redis API is really convenient, Upstash Redis has top-notch pricing model and awesome developer experience, and it’s fast. I love the combination!

AWS Lambda + Upstash Redis + Go = 🚀❤️

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Twitter 📝 Post-Commit Reviews by Cindy Sridharan]]> https://www.lambrospetrou.com/articles/twitter-post-commit-reviews/ twitter-post-commit-reviews Mon, 22 Aug 2022 00:00:00 GMT Check the original Twitter thread at @lambrospetrou/status/1561752140909002757.

Post-Commit Reviews - https://copyconstruct.medium.com/post-commit-reviews-b4cc2163ac7a

A great, and IMO controversial, article by @copyconstruct on why it’s better to code review after a change is merged into trunk. I have my opinions as well, so let’s dive in.

I spend a lot of time reviewing my team’s diffs, and I am a huge advocate for its benefits. So, this article caught me off-guard. After reading it, though, I see its perspective, but I still think that this can rarely work in practice.

I think the article should first start with the challenges of post-commit reviews. Right now, it paints a very nice picture at the beginning, but reading it to the end shows how hard it would be to actually implement.

👎 Challenge 1: High-Functioning, High-Trust Environments

Not only do the team members need to fully trust each other in shipping unreviewed code, but they also need to accommodate for onboarding new hires to this way of working.

👎 Challenge 2: Investment in Automation

Without great tooling and automation, post-commit reviews cannot work. It has to be trivial to revert commits, especially multiple ones. Doing this in monorepos is even more complicated.

managing commits becomes a lot easier in non-monorepo environments, since the automation required to revert commits doesn’t require research paper levels of complexity.

+1. This has been my experience as well.

👎 Challenge 3: Strong Cultural Scaffolding

This challenge is covered in a pro-point. Quoting the author:

“Getting to a point where post-commit reviews are a reality requires a strong cultural scaffolding. At the very least, it requires:

  • a culture of collaborating […] prior to code implementation via a design document.
  • consensus around aspects like style-guide, coding idioms, concurrency primitives etc.
  • investment in better automation practices and tooling.”

👍 Benefit 1: Focus

Being able to merge pull requests in quick succession ensures that the developer can iterate faster on the feature they are developing

Can’t argue with this.

Another benefit […] is reviewer focus. […] being pinged every 10 or 15 minutes for a review can hinder the productivity

This can be fixed. Open a PR, and continue working on the next PR. Or use a stacked-diffs tool like Phabricator.

👍 Benefit 2: Encourages Better Development Practices

This is the other side of the same coin as Challenge 3 above. Having a super disciplined team is required for post-commit reviews, so you get that as a benefit too.

👍 Benefit 3: Detect More Bugs Before Code Review

Undecided 🤔 I see how post-commit/pre-deploy testing can expose more issues, but I guesstimate them to be the minority. You should anyway test your code before sending it for review/merging.

A concern, fitting the trust point. With post-commit reviews, you need to trust your colleagues to actually go back and change big chunks of code if needed. I worked with a few people that would NEVER do that, and such cases are blockers.

In conclusion, I can see how post-commit reviews can speed up a team. But it requires all of:

  • great tooling
  • trust within the team
  • pre-commit decision-making processes (e.g. RFCs, design docs)

If you have these, go for it 🚀

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Twitter 📝 Observability for emerging infra: Charity Majors]]> https://www.lambrospetrou.com/articles/twitter-observability-for-emerging-infra-charity-majors/ twitter-observability-for-emerging-infra-charity-majors Sun, 14 Aug 2022 00:00:00 GMT Check the original Twitter thread at @lambrospetrou/status/1558819676439945216.

Just watched the “Observability for emerging infra” talk by @mipsytipsy. Even though I am a big proponent for testing before production, I do agree with all the points raised!

My favourite points raised 👇

1/ Testing is not exclusive to either staging or production. You can, and should, test in both.

2/ Shipping to production is not a flip switch that goes from 0 to 100 once the code lands. There can be commit reverts, hotfixes/patches, multiple versions running, automatic rollbacks. Our job is not done just because the code is shipped.

3/ If you can solve your problem with a LAMP-stack (or equivalent), do yourself a favor and go with it, ignoring all the K8s etc complexity.

4/ The shift from monitoring to observability is the shift from “known unknowns” to “unknown unknowns”. Monitoring is like unit tests, we know what to monitor. Distributed systems have an infinite list of failures that make staging worthless.

5/ We should be spending more time and money in tools for looking into production systems. We cannot test everything in staging environments, so we should admit that we need testing in production.

6/ Why do people invest so much in staging testing tooling, when they cannot tell if the system is healthy in the first place, staging or production. Without observability, it’s just chaos.

7/ Engineers should build muscle memory. When you ship code, you should go and look at it in production. We need to watch our code run with:

  • real data
  • real users
  • real traffic
  • real scale
  • real concurrency
  • real network

8/ What to test before prod, the known unknowns:

  • does it work
  • does my code run
  • does it fail in ways I can predict
  • does it fail in ways it has previously failed

9/ What to test in prod, the unknown unknowns:

  • complex behavorial tests
  • experiments (A/B testing)
  • load tests
  • edge cases/weird bugs
  • canary, canary, canary
  • rolling deploys
  • multi-region

10/ Risks of testing in prod

  • expose security vulnerabilities
  • data loss or contamination of hosts
  • sudden app crashes
  • resource saturation (due to load testing)
  • impossible rollback depending on error
  • bad experience to users

11/ We need to start using:

  • feature flags for continuous releases
  • high cardinality tooling (Honeycomb, FB’s Scuba)
  • canary
  • shadow systems
  • capture/replay for databases
  • Be less afraid, by using the right tooling

12/ Every engineer should know:

  • what normal production looks like
  • how to deploy and rollback to a known state
  • how to debug in production

13/ SSH-ing into production is a sign something is wrong with instrumentation!

14/ What is observability?

Control theory: It’s how much you can understand about the state of the system by looking at its external outputs.

Us: How can we ask new questions, new queries, without having to ship new code each time.

15/ Events, not metrics? Log strings vs structured events? We need high cardinality -> context, using structured data. Metrics and aggregations strip away the details.

16/ We need raw requests for new investigation queries, so use sampling instead. Aggregations are the devil, a once-way trip. Intelligent dynamic sampling can save space, but help us investigate issues.

17/ We need:

  • observability driven development
  • comfortably looking at prod
  • observability oriented tooling

Zero users care what the “system” health is, all users care about is their experience. So, watch it run in production.

That’s it! Great talk👌

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Fly.io cloud development environment with Visual Studio Code Remote-SSH]]> https://www.lambrospetrou.com/articles/flyio-cloud-dev-env/ flyio-cloud-dev-env Mon, 01 Aug 2022 00:00:00 GMT For many years now I have been using an AWS EC2 T3.nano (T2.nano previously) instance as my personal VPS server. I mainly host a Gitea server, along with some other toy projects, and I sometimes use it as my remote development machine when for some reason I cannot use the local laptop I have at hand. It’s been quite reliable honestly, checking its current uptime it says it has been up for 850 days now without any issues. And the reason I restarted it back then was to (unsuccessfully) upgrade from Amazon Linux to Amazon Linux 2.

Fast-forward to present, I find myself playing a lot with Fly.io, a new-ish cloud compute provider for server applications (e.g. Dockerfile). I covered it in a past article about serverless platforms in 2022, but in summary I really love the developer experience and the simplicity it provides through its CLI.

In this article I will describe how I now use Fly.io as my development environment in the cloud instead of EC2, without having to remember to update the underlying OS or even worse upgrading to a new major version and having to do annoying file migrations.

It’s all containers

Fly.io is built on top of Firecracker microVMs and supports a few types of builders that ultimately assemble a container to deploy. In this article I am going to use the Dockerfile support since that’s how I prefer to model my development environment.

The benefits of modelling my environment in a Dockerfile:

  • Upgrading the operating system (OS), or even changing it, is a single line change and a redeploy (e.g. from FROM ubuntu:18.04 to FROM ubuntu:20.04).
  • Installing or uninstalling software and packages from the OS is again trivial using the RUN command.
  • Define exactly what processes should my server run and never have to worry about manually (re)starting them.
  • Have all of the above versioned in Git, so doing any change to it is easy while keeping a historical record in case I need to check how something was setup in the past.
  • Have the whole environment recreated in seconds if something goes wrong (Fly.io provides persistent disk volumes).

Cloud Development Environment

For a basic remote development environment I want to be able to do the following:

  • SSH into it in case I need to test linux commands.
  • Checkout and work with Git repositories.
  • Install the several programming languages I work with.
  • Have a nice development experience using Visual Studio Code comparable to local development.

The full source code referenced in the following sections is available in this repository, and you can download it as .zip file here.

Dockerfile

This is a simplistic version of the Dockerfile I use:

FROM ubuntu:bionic

RUN apt-get update && apt-get install --no-install-recommends -y \
    ca-certificates curl sudo openssh-server bash git \
    iproute2 apt-transport-https gnupg-agent software-properties-common \
    # Install extra packages you need for your dev environment
    htop make vim && \
    apt autoremove -y

ARG USER="clouddevuser"
RUN test -n "$USER"
# Create the user
RUN adduser --disabled-password --gecos '' --home /data/home ${USER}
# passwordless sudo for your user's group
RUN echo "%${USER} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

ENV USER_ARG=${USER}

# Setup your SSH server daemon, copy pre-generated keys
RUN rm -rf /etc/ssh/ssh_host_*_key*
COPY etc/ssh/sshd_config /etc/ssh/sshd_config

COPY ./entrypoint.sh ./entrypoint.sh
COPY ./docker-entrypoint.d/* ./docker-entrypoint.d/

ENTRYPOINT ["./entrypoint.sh"]
CMD ["/usr/sbin/sshd", "-D"]

Let’s do a simple breakdown of each Dockerfile instruction.

First, we define the Operating System (OS) to use, in this case Ubuntu version bionic (codenamed 18.04).

FROM ubuntu:bionic

Then we encounter the RUN instruction which basically installs additional software to our OS. We can update this list (see relevant line comment) to include more packages as necessary.

The next 8 lines create a new user clouddevuser (customisable argument using ARG), sets the necessary permissions, and creates the user’s home under /data/home (we can change this to /home/clouddevuser or anything else). The thing to remember from these lines is that we export an environment variable USER_ARG which holds the user name since it will be used by the entrypoint.sh script (detailed in the next section).

Then, we have 2 lines essentially making sure the our local SSH configuration is copied into the container and is the one used.

The last lines need some explanation.

COPY ./entrypoint.sh ./entrypoint.sh
COPY ./docker-entrypoint.d/* ./docker-entrypoint.d/

ENTRYPOINT ["./entrypoint.sh"]
CMD ["/usr/sbin/sshd", "-D"]

The two COPY instructions copy the local entrypoint.sh script and the local directory’s ./docker-entrypoint.d/ content to the container’s root directory. The last pair of instructions specify that we want our container to execute ./entrypoint.sh /usr/sbin/sshd -D using the ENTRYPOINT and CMD Dockerfile instructions.

In summary, this Dockerfile selects the OS we want, installs extra software packages, sets up our user, and then runs the entrypoint.sh script. Easy.

entrypoint.sh

#!/bin/bash

set -euxo pipefail

echo "Creating /run/sshd"
mkdir -p /run/sshd

HOME_DIR=/data/home
SSH_DIR=/data/etc/ssh

echo "Ensure home directory"
mkdir -p $HOME_DIR

echo "Ensure SSH host keys"
mkdir -p $SSH_DIR
ssh-keygen -A -f /data

echo "Setup SSH access for user $USER_ARG"
mkdir -p $HOME_DIR/.ssh
# Append the given keys to the authorised keys and only keep the uniques!
# The `# empty comment` is to avoid an empty file which causes grep to fail.
echo -e "# empty comment\n$HOME_SSH_AUTHORIZED_KEYS" >> $HOME_DIR/.ssh/authorized_keys
cat $HOME_DIR/.ssh/authorized_keys | sort | uniq | grep -v "^$" > /tmp/authorized_keys
mv /tmp/authorized_keys $HOME_DIR/.ssh/authorized_keys

if [ -f "$HOME_DIR/.ssh/id_ed25519" ]; then
    echo "$HOME_DIR/.ssh/id_ed25519 exists, skipping."
    echo ""
    echo "Make sure you add this public key to your Github / Gitlab / other vcs:"
else
    echo "$HOME_DIR/.ssh/id_ed25519 does not exist, generating."
    ssh-keygen -t ed25519 -f $HOME_DIR/.ssh/id_ed25519 -C "$USER_ARG@fly-vscode" -N ""
    echo ""
    echo "Add this public key to your Github / Gitlab / other vcs:"
fi
cat $HOME_DIR/.ssh/id_ed25519.pub

echo "chowning your home to you"
chown -R $USER_ARG:$USER_ARG $HOME_DIR

if [[ -d "docker-entrypoint.d" ]]
then
    echo "Running docker-entrypoint.d files"
    /bin/run-parts --verbose docker-entrypoint.d
fi

echo "Running $@"
exec "$@"

Briefly, the entrypoint script:

  1. creates the user’s home directory (needs to match the one we used in Dockerfile)
  2. adds some SSH authorized keys to our configuration to allow remote SSH-ing
  3. optionally generates an SSH key that we will use to authenticate this server with services like Github when pulling/pushing Git repositories
  4. it runs all the scripts under the /docker-entrypoint.d/ directory
  5. and finally runs the command passed as argument to the script, which as we explained in the previous section it’s going to be /usr/sbin/sshd -D

Notes

  • Anything that needs the user uses the environment variable USER_ARG that we defined in the Dockerfile.
  • The content of the environment variable HOME_SSH_AUTHORIZED_KEYS is appended to the $HOME_DIR/.ssh/authorized_keys file, and then we do some shell shenanigans to only allow unique lines inside that file (to avoid appending the same keys each time our container is started). The value of HOME_SSH_AUTHORIZED_KEYS is provided by the fly.toml file as you will see below.
  • The generated $HOME_DIR/.ssh/id_ed25519.pub file is the public key we need to upload to Github or any other service that needs to authenticate its user using SSH keys.
  • The /bin/run-parts --verbose docker-entrypoint.d conveniently runs any script we put into the docker-entrypoint.d/ directory, which makes it easy to do any initialisation to the OS at runtime.

After the above script runs, now the container is stuck in the /usr/sbin/sshd -D command. This is the SSH daemon that accepts connections from anyone attempting to SSH into our Fly.io server instance, our container. As long as the only thing we want to expose from the server is the SSH port, this is the only thing we need to run as the last command in our entrypoint.sh.

if we didn’t run this at the end, and the script exited, then the container would complete and exit as well, which would then cause our Fly.io server instance to be marked as failed/unhealthy and would be restarted, leading to a crash-loop (by default it attempts to deploy up to 3 times).

fly.toml

app = "<WILL_BE_REPLACE_WITH_GENERATED_NAME>"

[env]
  HOME_SSH_AUTHORIZED_KEYS = '''
'''

[[mounts]]
  # This is the persistent volume mount location, so if you change this
  # you need to also change the Dockerfile and entrypoint.sh wherever "/data" is used.
  destination = "/data"
  source = "clouddevdata"

[[services]]
  internal_port = 22
  protocol = "tcp"

  [[services.ports]]
    port = 10022

Fly.io uses the fly.toml configuration file to configure your application when using the flyctl CLI.

What happens here?

  • The app key specifies the name of the Fly.io application after it’s created (see below section) and is used by the flyctl CLI when issuing commands.
  • The HOME_SSH_AUTHORIZED_KEYS key can be updated to contain the SSH keys to put into the authorised keys file for our container at runtime. I usually update this with a new SSH key, deploy the application (see section below) which will append it to the authorized_keys file, and then remove it from the fly.toml file to avoid versioning it in the Git repository.
  • The clouddevdata persistent volume is mounted on /data inside the container.
  • We expose port 10022 publicly and route that to port 22 in the container, which is where the SSH daemon we started in our entrypoint script listens.

Enough! Give me my cloud environment.

OK, after we explained the key parts of the setup, let’s see how trivial it is to run this with Fly.io.

0. Get the code

The full source code is available in this repository, and you can download it as .zip file here.

All flyctl commands shown below need to run inside the source code directory.

1. Create a Fly.io application (once)

This step only needs to be run once in order to create your Fly.io application. After [installing the flyctl CLI](https://fly.io/docs/flyctl/installing/, run the following:

flyctl launch --generate-name --no-deploy --copy-config

Running the above will give you a prompt to select the region you want to deploy your cloud environment. After selecting the region, the fly.toml will also be updated with the autogenerated application name (as specified by --generate-name).

If you want to use a specific application name (since it’s part of the DNS name you will need to use), then you can use the --name argument instead (e.g. --name lambros-application-1), but keep in mind that this should be unique across all Fly.io applications globally since it’s part of the DNS subdomain you get, e.g. lambros-application-1.fly.dev, so there is a high chance your wanted name is taken.

2. Create the persistent volume (once)

The main enabler for the cloud development environment is that we can use persistent disk volumes to hold our data files while keeping all the OS/packages controlled by the Dockerfile. So let’s create our volume:

flyctl volumes create clouddevdata --region lhr --size 10

This will create a 10GB volume in London (lhr). You have to create the volume in the same region you selected in the previous step for the application itself! You can also check the available regions list to find the right value.

3. Deploy

This is the only step we need to do every time we change something in our application.

flyctl deploy

This will pick up the Dockerfile, check if there are changes and build a new image if necessary, and then trigger a deployment. Once the deployment is finished you can use the cloud development environment, i.e. SSH into it.

4a. Generate SSH keys

If you already have your SSH keys you can skip this section and go to Section 4b.

To generate your key run the following (replace the KEY_FILENAME and the email as necessary):

ssh-keygen -t ed25519 -f ~/.ssh/<KEY_FILENAME> -C "your_email@example.com"

If your system does not support the Ed25519 algorithm, you can use RSA keys.

ssh-keygen -t rsa -b 4096 -f ~/.ssh/<KEY_FILENAME> -C "your_email@example.com"

The above command will generate two files:

  1. The private key, at ~/.ssh/<KEY_FILENAME>, which should never be shared with anyone.
  2. The public key, at ~/.ssh/<KEY_FILENAME>.pub, which is the one to upload in Github or in our case paste in the HOME_SSH_AUTHORIZED_KEYS section as described above.

Common key filenames are id_<algorithm>, e.g. id_rsa, or id_ed25519. In some cases, I generate keys for different purposes so having specific names for the keys is very useful.

You should add the key to the ssh-agent for easier use, following the Github instructions.

4b. SSH

The default configuration only allows SSH-ing with authorized keys, and password-based authentication is disabled (see etc/ssh/sshd_config in the source code). Therefore, you need to update the HOME_SSH_AUTHORIZED_KEYS value in fly.toml with your laptop’s SSH key (usually ~/.ssh/id_rsa.pub), and then deploy once with flyctl deploy. Then, you can remove the SSH key from the HOME_SSH_AUTHORIZED_KEYS again to keep it safe.

Test that you can SSH into the cloud development environment (assuming application name lp1111, and user clouddevuser):

ssh clouddevuser@lp1111.fly.dev -p 10022

Now you should be hopefully logged into the remote container, so go nuts, and explore what you can do, do some file changes, and confirm that your changes persist across deployments 🤩

5. Visual Studio Code - Remote SSH

VSCode has amazing remote development capabilities which enable full-use of the editor with any remote server accessible over SSH connection.

  1. Install the Remote - SSH extension from the marketplace.
  2. The default configuration only allows SSH-ing with authorized keys, and password-based authentication is disabled (see etc/ssh/sshd_config in the source code). Therefore, you need to update the HOME_SSH_AUTHORIZED_KEYS value in fly.toml with your laptop’s SSH key (usually ~/.ssh/id_rsa.pub), and then deploy once with flyctl deploy. Then, you can remove the SSH key from the HOME_SSH_AUTHORIZED_KEYS again to keep it safe.
  3. Open the command palette (CMD+SHIFT+p on Mac, CTRL+SHIFT+p on Windows).
  4. Search for the Remote-SSH: Connect to Host command and select it.
  5. Type your environment details, e.g. clouddevuser@lp1111.fly.dev:10022, where clouddevuser is the user used in the Dockerfile, lp1111 is the Fly.io application name, and 10022 is the port exported by the Dockerfile.

You should then be able to mount any directory on the remote server and use VSCode as if it’s working with the local filesystem 🥳

Conclusion

Fly.io + VSCode Remote-SSH = ❤️ 🚀

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[AWS Elastic Beanstalk Go platform (AL2) — Deep Dive]]> https://www.lambrospetrou.com/articles/elastic-beanstalk-al2-go/ elastic-beanstalk-al2-go Sun, 30 Jan 2022 00:00:00 GMT Even though my default and preferred way of deploying applications is #serverless platforms, sometimes I need an actual long-running host. For example, some of the applications I run (e.g. Gitea) are written in Go and use SQLite as their database so in that case serverless does not work.

In this article I will explore the Elastic Beanstalk Go platform based on Amazon Linux 2 (AL2) for deploying a simple application. As a comparison, a couple of years ago I wrote about the older Go platform based on Amazon Linux 1.

Even though I focus on the Go platform, almost everything applies to all Amazon Linux 2 based platforms (e.g. Java). You can also use the Go platform to deploy any application that is compiled to a binary that can run on Amazon Linux 2 (e.g. Rust).

Disclaimer: All information shown is accurate as of Jan 30/2022.

Why use AWS Elastic Beanstalk?

Easy to begin, Impossible to outgrow.

Elastic Beanstalk is a Platform-as-a-Service (PaaS) service offering by AWS. It takes care of managing the platform updates, patching the underlying hosts, and abstracting away the AWS resources used while at the same time providing the necessary hooks and capabilities to customise and extend them when needed.

AWS Elastic Beanstalk is similar in “principle” to Azure App Service, or Google App Engine.

I personally like Elastic Beanstalk and is my first choice of deploying to actual hosts/VM instances. I usually use the single instance environment to get the benefits without the costs, but there is also the load-balanced environment which allows you to scale as much as you want.

Features the Go platform provides which I cannot find in another platform as of time of writing (see related tweet):

  • single server instance only (because not everything is web-scale)
  • persistent disk volumes (i.e. Amazon EBS)
  • deployments in-place on same instance with only 1-2s of downtime
    • restriction due to the persistent disk volume only being accessible from one host at a time
  • managed platform: OS updates, host patching, language runtime updates
  • variety of instance types, from the low-cost T3, to the powerful ARM-based Graviton2 C6g
  • <5$/month

Overview

The Amazon Linux 2 based platforms were added back in 2020, and with that upgrade all the platforms now follow (almost) the same conventions and setup (see supported Linux platforms), which is a nice improvement over the previous platforms based on Amazon Linux 1 where each language had its own idiosyncracies.

The AWS documentation contains most of the things we need to know about the platforms but unfortunately as usual it’s like they intentionally try to make it hard on their users, and they spread the information across multiple pages. In this article I will try to cover the main features of the platform and link to the corresponding documentation pages to make it easier to find what we need. This is basically a reference for future me 😅

All AL2 platforms offer the following features (see documentation):

  • Buildfile: one-off commands to run from inside the uploaded bundle, e.g. building the source code to create the binary to run.
  • Procfile: long-running monitored commands to run, e.g. the web server.
  • Platform hooks: one-off commands to run during the deployment lifecycle hooks (e.g. predeploy, postdeploy). New to AL2 based platforms.
  • Reverse proxy configuration: nginx is used as the reverse proxy for most platforms and there is a way to override its configuration. Some platforms also provide Apache HTTPD as proxy (e.g. Tomcat, Node.js, PHP, and Python).
  • Configuration files (.ebextensions): configuration to extend the Beanstalk environment. For example, add new CloudFormation resources (e.g. DynamoDB tables, EBS volumes), customise existing CloudFormation resources (e.g. the EC2 instance), override files on the filesystem, and many more.

Let’s go into details for each one of these for the Go platform specifically.

Buildfile

The Buildfile file should be placed at the root of the uploaded bundle .zip file and contains one-off commands that we want to run as part of the build phase of the deployment. The working directory of each command is the root of the uploaded bundle. If your uploaded bundle contains your source code then this is where you would put your application compile command. If you have a Continuous Integration (CI) system to build your source code before-hand then you don’t need a Buildfile.

For example, the following Buildfile will compile the Go application and create a binary file named app into the bin/ directory:

cmd1: go build -o bin/app

The cmd1: part is nothing but just a name and does not have any meaning other than differentiating the commands to run. The following is also valid:

build: go build -o bin/app
check: ./validate-app-binary.sh

Personal opinion: I never use the Buildfile to build my application and I always have a CI system to build the source code (or even build it locally). Even the documentation itself suggests to use the predeploy platform hook (more later) instead for the one-off commands we want to run, so this is not really needed. This means that my uploaded bundle .zip file only contains the compiled artifacts, e.g. the bin/app binary from the above example. This makes deployments a bit faster as well since the build phase will not happen during the Beanstalk lifecycle and I don’t worry about my build process being constrained by the machine running the actual application. For example, if you have a lot of dependencies or your tests need a lot of resources then doing it before triggering a Beanstalk deployment will allow you to use smaller lightweight instances for the actual application in production (e.g. T3.nano, T4G.nano).

Procfile

The Go platform also supports conventions to automatically build and run your application, but I personally avoid them since they are less flexible than using Procfile and there is zero benefit in using them.

The Procfile file should be placed at the root of the uploaded bundle .zip file and contains the long-running commands that we want to run as part of the application. Each process started by these commands will be monitored by the environment and automatically restarted when they crash. This is the place where we specify the main command(s) to start the application.

For example, the following starts the bin/app and bin/bgapp apps with the right flags:

web: ./bin/app -name webapp
bgapp: ./bin/bgapp -name bgapp -port 5001

By default, Elastic Beanstalk listens to requests from the internet on HTTP port 80 and forwards them to port 5000 inside the host (configurable if needed). Therefore, the application should listen on port 5000. Elastic Beanstalk looks for the command named web from Procfile, and sets the PORT=5000 environment variable when running it. The application code will need to read that env variable when setting up its HTTP listener. We could also ignore the PORT env variable and hardcode port 5000 but it makes it harder to start your application on different ports (e.g. locally), so do the nice thing and read the env variable on startup (see example application code below).

NOTE1: The default port can be changed from 5000 to something else by using .ebextensions and setting the aws:elasticbeanstalk:application:environment.PORT property. Read more about this in the reverse proxy configuration docs (make sure to expand the “Reverse proxy configuration” section).

NOTE2: In the previous version of the platform (AL1) all processes in the Procfile were getting the PORT set to a value starting from 5000 and then growing in increments of +100. This does not apply anymore in AL2 platforms and only the explicitly named web process has the PORT env set, which is why I explicitly set the port for bgapp in the above example.

NOTE3: There are some nuances around logging and how the logs are streamed to CloudWatch Logs when Procfile contains more than the web process but we will explore those later.

Platform hooks

Platform hooks are a new addition to the AL2 based platforms. The only way to extend and customise Elastic Beanstalk before was using the .ebextensions configuration files but that was/is complicated and very error-prone (lots of trial-and-error needed 😓). Platform hooks cover 90% of all use-cases (personal guesstimate), and they make our lives much easier.

The Platform hooks (expand section) are basically script files placed inside the .platform/hooks directory, under one of the following sub-directories: prebuild, predeploy, postdeploy, depending when they need to run.

The scripts inside each of those directories run in order of their filename. For example, let’s examine the following directory structure:

$ ls .platform/hooks/**
.platform/hooks/postdeploy:
01-validate-http.sh  02-update-route53.sh

.platform/hooks/predeploy:
01-setup-ebs.sh

With the above files the script 01-setup-ebs.sh will be executed before the new version of the app is activated (predeploy), and the 01-validate-http.sh and 02-update-route53.sh scripts will run after the switch (postdeploy) in this specific order.

Config hooks

Apart from the .platform/hooks directory, there is also a .platform/confighooks directory supported. At first glance, the difference between the two is confusing. Even though the available hooks and execution rules are the same, the type of changes in a deployment will decide which hooks to trigger.

A configuration deployment occurs when you make configuration changes that only update environment instances without recreating them.

According to the docs the following changes trigger .platform/confighooks only:

  • Environment properties and platform-specific settings
  • Static files
  • AWS X-Ray daemon
  • Log storage and streaming
  • Application port

NOTE: In order to make sure a script runs regardless of the type of changes it has to be placed in both .platform/confighooks and .platform/hooks.

Reverse Proxy configuration

All AL2 based platforms now are similar in the way their reverse proxy is configured, using nginx. In order to customise the reverse proxy with our own nginx configuration we can use the .platform/nginx directory. Any .conf configuration file inside the directory .platform/nginx/conf.d/ of the uploaded application source bundle will be included automatically by nginx during service startup.

I rarely have to modify the proxy configuration, and when I do it’s to adapt some of the default limits, e.g. concurrency connections, timeouts. See nginx examples for what configuration is available, and the Multiple Procfile processes section below for how this can be used to proxy two servers running on the same host.

Advanced configuration files (.ebextensions)

I am not going to dive deep into this one because it supports a massive amount of configuration, and should be the last option only if nothing else works. Configuration files are useful when all the above features we discussed (Buildfile, Procfile, .platform/{confighooks/hooks}) do not support what we want. In this case they can be handy since they support almost anything.

  • A configuration file needs to be placed inside the .ebextensions directory at the root of the uploaded .zip file bundle, and should have the .config file extension.
  • The configuration file structure is documented here and supports anything from installing yum packages, to putting files on the filesystem, to adding/overriding CloudFormation resources.
  • Read the above documentation pages in order to properly understand how .ebextensions work to avoid a lot of unnecessary head wall-hitting 🤕

Example of a configuration file (.ebextensions/awslogs.config) that configures the CloudWatch Logs agent to stream the /var/log/bgapp.stdout.log log file to CloudWatch Logs: https://gist.github.com/lambrospetrou/758a312a1317532eb6bb0960985df83e

Instance deployment workflow

The documentation section I visit most often is the instance deployment workflow diagram.

Unfortunately, it’s not 100% complete since it’s not showing in detail how shutdown and termination of the current app is happening, but it covers everything else.

Credits for the following image belong to AWS but I am putting it here for completeness as well:

AL2 platform instance deployment lifecycle

Example application

In this application we have a simple HTTP API written in Go. We don’t use any of the advanced configuration features of Elastic Beanstalk here just to show that things can be easy and simple.

I didn’t put a Buildfile in this application, so when we want to deploy we need to run make either locally or on our CI system, and then upload the bundle file located in build/bundle.zip.

This is how the directory structure looks like after running make:

➜  single-process git:(master) find . -type f
./app/go.mod
./app/main.go
./Makefile
./build/bin/app
./build/bundle.zip
./build/Procfile
./build-tools/Procfile

Procfile

web: ./bin/app -name web

Application code (main.go)

package main

import (
    "flag"
    "fmt"
    "html"
    "log"
    "net/http"
    "os"
    "strings"
)

func main() {
    name := flag.String("name", "app", "The name of the service running, e.g. web2")
    port := flag.String("port", os.Getenv("PORT"), "The port for the server to listen.")
    flag.Parse()
    if strings.TrimSpace(*port) == "" {
        *port = "5000"
    }

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        msg := fmt.Sprintf("Service %s Path, %q", *name, html.EscapeString(r.URL.Path))
        log.Println(msg)
        fmt.Fprintf(w, msg)
    })
    log.Printf("App %s starts listening at :%s\n", *name, *port)
    log.Fatal(http.ListenAndServe(":"+*port, nil))
}

Notice how we read the PORT environment variable but also support command line arguments and then start our HTTP listener on the given port.

After deploying the above on Elastic Beanstalk and enabling CloudWatch Logs streaming we can see that automatically we get the following Log groups created and populated:

single instance CloudWatch Logs

The /aws/elasticbeanstalk/Al2go-env-1/var/log/web.stdout.log log group contains the log lines from our application. The web prefix of the file corresponds to the web process name inside our Procfile.

The /aws/elasticbeanstalk/Al2go-env-1/var/log/eb-engine.log log group contains all logs from the Beanstalk processes that run during a deployment, which is useful while troubleshooting failed deployments or trying to understand how deployments work.

Elastic Beanstalk provides an endpoint for our application which corresponds to the HTTP port 80 mentioned in Procfile section above, and looks like http://al2go-env-1.eba-f2qnm2t6.eu-west-1.elasticbeanstalk.com/ where al2go-env-1 is the name of my environment. Unfortunately this endpoint is not HTTPS…

Multiple Procfile processes

In this section we have the same application as above, but this time I am starting it twice listening on different ports in order to show how we can override nginx to proxy two applications on the same host.

This is how the directory structure looks like after running make:

➜  multi-process git:(master) find . -type f
./app/go.mod
./app/main.go
./Makefile
./.ebextensions/awslogs.config
./build/bin/app
./build/bundle.zip
./build/.ebextensions/awslogs.config
./build/Procfile
./build/.platform/nginx/conf.d/01_proxy.conf
./build-tools/Procfile
./.platform/nginx/conf.d/01_proxy.conf

Procfile

web: bin/app -name web
bgapp: bin/app -name bgapp -port 5001

Application code (main.go)

Exactly the same as before.

What happens?

We have a process named web which is the same as before starting the server at given PORT (5000), and also a second process named bgapp which starts the same server at port 5001 explicitly. As I mentioned in the Procfile section only the server at port 5000 will actually receive requests by default, so we need to override the nginx configuration to route traffic accordingly.

We do this by providing .platform/nginx/conf.d/01_proxy.conf:

server {
    server_name .elasticbeanstalk.com;
    listen 80;

    location /web2 {
        proxy_pass http://127.0.0.1:5001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

When Beanstalk starts the nginx service it automatically includes all .conf files in .platform/nginx/conf.d/ and therefore our configuration above will be used. What it does is basically route any request under the /web2 path to our server named bgapp, and anything else goes to the main server named web (remember our Procfile).

Logging

Unfortunately Elastic Beanstalk has some rough edges, and logging is one of those. When we only have a single web process everything is fine and as we saw above all the logs we need are automatically streamed to CloudWatch Logs (if enabled). The issue is when we have more than one process, e.g. in our current scenario where we also have the bgapp process.

After reading the following resources I realised I need to provide a custom configuration file to install AWS CloudWatch Logs agent and set it up to stream the additional bgapp.stdout.log file to CloudWatch Logs.

You can find the .ebextensions/awslogs.config file I used at https://github.com/lambrospetrou/aws-playground/blob/master/elastic-beanstalk-al2-go/multi-process/.ebextensions/awslogs.config.

NOTE1: The default IAM role created by Elastic Beanstalk for web services, AWSElasticBeanstalkWebTier, does not include the logs:CreateLogGroup permission which is needed for the creation of the /var/log/bgapp.stdout.log log group so make sure to update the IAM role used by the instance to include that. The default log groups (e.g. for /var/log/web.stdout.log) are created by the Elastic Beanstalk service automatically hence why the role used by the EC2 instance does not (need to) have this permission. But since we are now adding custom logs to be streamed to CloudWatch Logs we need to attach explicit permissions to allow the creation of the log group as well.

NOTE2: I opened a roadmap feature request to the Beanstalk team to make this automatically for all processes inside the Procfile since it doesn’t make any sense to me to have to do this on my own since it’s already done for the web process.

Conclusion

AWS Elastic Beanstalk is simple, and flexible enough to do anything I want. When AWS Lambda cannot be used, this is my goto platform.

In this article I put everything I had in my notes from using Elastic Beanstalk in my own projects over the years. There are some rough edges but the main issue I encounter is that the documentation is spread all over the place. Having this information structured makes it a bit easier to use and troubleshoot things.

In the worst case, even if nobody else needs this, future me will be very happy I took the time to write this down 😎

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Serverless platforms into 2022 — AWS, Cloudflare Workers, Netlify, Vercel, Fly.io]]> https://www.lambrospetrou.com/articles/serverless-platforms-2022/ serverless-platforms-2022 Thu, 20 Jan 2022 00:00:00 GMT I love #serverless platforms 🚀 I have used most of the public offerings over the years, and I always keep an eye out for new services. In this article I explain the different types of compute, edge vs regional, and make a comparison among the most popular providers. I know there are more providers out there offering serverless products but I included the ones I personally tried. Anything else can be mapped to one of these anyway for comparison.

Disclaimer: All data shown below is as of Jan 20, 2022.

Compute @ Edge vs Regional

There are a few confusing terms floating around, not uncommon for our industry unfortunately, so let’s try to clarify things. One of the most important differences among the different platforms is their answer to the question where does my code run?

Check Amazon CloudFront’s presence map:

Amazon CloudFront presence map

Looking at the legend, and then the map, we can immediately see that the most important distinction between Edge locations and Regions is how many there are of each. We have 300+ Edge locations, or Points of Presence (PoP), and 13 Regional Caches (bigger orange circles). The Regional Caches are mostly backed by standard AWS regions.

For comparison let’s look at Cloudflare’s global map of more than 250 edge locations:

Cloudflare presence map

So, what’s the main difference between an edge location, and a regional location? Well, there are a lot more edge locations, and they are more spread around the world. This means that users are more likely to be closer to an edge location rather than a regional location.

  • Compute - Edge: When platforms say they offer edge compute, the reasonable assumption is that they run our code in all of these edge locations in their underlying CDN infrastructure.
  • Compute - Regional: When we have the traditional regional compute it means that we decide which region will run our code, and then all users have to reach that single region.
  • Compute - Multi-regional: We also have the middle ground where our code runs in the regional locations, but we don’t explicitly specify which one. This implies that the provider will run our code in the region that is the closest to the user making the request.

The selling point of running on an edge compute platform is that our customers will get faster response times since our code runs closer to them. However, keep in mind that if the code needs access to a database or calls into external services, it means that those services are now a bottleneck and add to the overall latency. Everything has tradeoffs and the right choice depends on several factors.

Now that the terms are understood, let’s see how the different platforms compare.

The platforms

In this section I will briefly enumerate all the platforms, and what they offer. AWS has multiple serverless products, each with different features, so I will list them separately.

AWS Lambda - Regional

Run code without thinking about servers or clusters

The service that started it all in 2014! Easily the most popular serverless product, and with the most available integrations both inside the AWS ecosystem, but also with other SaaS products. AWS Lambda over the years got many features, and at the moment it supports many runtimes.

  • Languages (natively): Go, Node.js, Python, Ruby, Java, C#, PowerShell
  • Languages (custom runtime): Any language as long as you implement the custom runtime API (since 2018)
  • Containers: Any Docker container implementing the Runtime API, Extensions API, and Logs API (since 2020)
  • Memory: 128MB up to 10GB
  • Runtime: 29s (HTTP integration with API Gateway), 15min (anything else)
  • Uploaded Bundle size: 50MB (zipped), 250MB (unzipped), 10GB (container image)

Looking at the above features, it’s quite clear that AWS Lambda can run pretty much anything now 🚀 But you should be aware of the cold starts…

AWS Lambda@Edge - Multi-regional

Run your code closer to your users

Lambda@Edge launched soon after Cloudflare Workers (see below) came out so many consider this a rushed response to Workers. Its goal is to run our code in the region closest to the user making the request. Therefore, despite its name containing the word edge, it’s not an edge compute product as per our above definition. Yes, this definitely has many people fooled.

Lambda@Edge is a product that can be used only with a CloudFront distribution, and you specify which events to handle while the request flows through the CloudFront cache systems, e.g. before checking the cache, or after the origin returned a response. Read the docs for details about the different integration triggers.

  • Languages: Node.js, Python
  • Memory: 128MB (Viewer triggers), 10GB (Origin triggers - same as AWS Lambda)
  • Runtime: 5s (Viewer triggers), 30s (Origin triggers)
  • Uploaded Bundle size: 1MB (Viewer triggers), 50MB (Origin triggers)

I have used Lambda@Edge extensively while working at Amazon/AWS and it’s great. The 1MB bundle size limitation was the only issue we bumped into until we minified our JavaScript.

Amazon CloudFront Functions - Edge

lightweight functions in JavaScript for high-scale, latency-sensitive CDN customizations

CloudFront Functions launched last year in 2021 as one yet another competitor to Cloudflare Workers, but yet again a limited one in my opinion. CloudFront Functions is a proper edge compute product, since our code runs in all of CloudFront’s edge locations. I do use it in my own website (the one you are reading now) and it does the job, but its runtime restrictions limit the potential use-cases, especially since you are not allowed to make any external API calls or even access a filesystem.

  • Languages: Custom JavaScript runtime
  • Memory: 2MB
  • Runtime: 5s (Viewer triggers)
  • Uploaded Bundle size: 10KB

It’s clear this is a pretty restricted environment, which to be fair is implicitly acknowledged by AWS judging by the use-cases they list as ideal for the product.

Cloudflare Workers - Edge

Deploy serverless code instantly across the globe to give it exceptional performance, reliability, and scale.

Workers was one of the first true edge compute offerings and pushed the whole industry forward. The main differentiator is that Workers run in V8 isolates eliminating cold starts completely, and allowing near-instantaneous execution very close to users. Note that the V8 environment is not a full Node.js runtime which means it’s not possible to run everything.

  • Languages: JavaScript/TypeScript, WebAssembly
  • Memory: 128MB
  • Runtime: 50ms (Bundled Plan), 30s (Unbundled Plan - HTTP), 15min (Unbundled Plan - Cron trigger)
  • Uploaded Bundle size: 1MB

One important feature of Cloudflare Workers is that the runtime is measured in CPU consumption for the Bundled Usage plan, meaning that external API requests (fetch requests) do not count towards the limit. However, for the Unbundled Usage plan the runtime is measured in wall-clock time, so we are charged for the whole duration of the worker running, including external calls.

The biggest advantage of Workers over its competitors is that it’s extremely nicely integrated with the rest of Cloudflare. It natively supports accessing and updating the CDN cache, it provides Durable Objects which are very powerful, and it has a Key-Value store built-in.

Another recent addition is the native integration of Workers with Cloudflare Pages which enable full-stack application development completely on Cloudflare. With Cloudflare R2 Storage around the corner this is going to be a very big threat to AWS Lambda dominance.

Netlify / Vercel - Edge + Regional

Develop. Preview. Ship.

Netlify and Vercel are the most popular serverless products among the frontend community. Their focus is entirely on improving the developer experience for frontend developers, and they are doing an incredibly amazing job. Vercel standardised the phrase Develop. Preview. Ship. and Netlify straight up competes with them at that.

I have been using both of them on-and-off for several years and honestly feature-wise they are identical. From my perspective the differences between them are in their satellite (or secondary) features. Vercel focuses on extending their platform’s features (e.g. Next.js Live), and adding native optimisations for Next.js (their amazing React-based framework) e.g. Image Optimization. Netlify on the other hand provides several features that are complementary to a frontend application, like Identity, Forms, Split Testing and more.

The interesting fact about these platforms is that they are built on-top of the products I went through above.

Their serverless functions products (Netlify, Vercel) are directly build on-top of AWS Lambda, and their edge functions are built on-top of Cloudflare Workers (Netlify, Vercel). Weirdly though, Netlify lists their edge handlers memory limit as 256MB, whereas Cloudflare Workers tops at 128MB, so I am not sure what’s going on there, if there is a special agreement between them, or if they actually run these on their own network!

So, are Netlify and Vercel the best products to use? Should everyone migrate their AWS (or other) serverless apps to them since they combine both? Well… Nope 😅

These products provide top-notch developer experience but it comes at a cost. Their serverless offerings are more limited than their native counterparts:

  • Serverless functions:
    • Netlify: only us-east-1 region, 1GB of memory, 10s synchronous execution
    • Vercel: only IAD1 region, 1GB of memory, 5s synchronous execution (15s on Pro, 30s on Enterprise)
  • Edge functions:
    • Netlify: 256MB of memory, 50ms runtime
    • Vercel: 1.5s runtime (after returning a response it can still run up to 30s)

Some of the above limits can be lifted for their paid plans, but mostly the Enterprise ones.

Cloudflare Pages gets an honorary mention in this section since it now competes in this space by offering a seamless integration with source code repositories, and serving Jamstack websites, while at the same time having built-in integration with Cloudflare Workers.

The above limitations are why I personally still use AWS Lambda directly when I want an API, and combine it with Cloudflare Pages for the frontend. However, if your application does not need more resources than what these platforms offer, which to be fair most websites don’t, then you are fine just using them and focusing on your application. This is exactly what I did when I built Minibri Temp with Netlify.

Fly.io - Multi-regional

Fly keeps your performance up by sending users on the shortest, fastest path to where your application is running.

Fly.io is a relatively new player (launched in 2020) but it definitely attracted many people already. It currently has 20 regions and the selling pitch is that you specify a Docker image for your application and depending on where the user is they will spin up an instance of it to the closest region and serve the request. Very similar to how Lambda@Edge works, but incredibly more flexible!

  • Containers: Docker images, Cloud Buildpacks
  • Memory: 256MB-2GB (shared-cpu), 2-64GB (dedicated-cpu)
  • Runtime: No limit, pay per second
  • Uploaded Bundle size: n/a

This is more like a traditional container product, but I included it because they do offer the multi-regional flavor, and they even have multi-region Postgres databases with a nice architecture. If their scaling is fast enough not to have long cold-starts then this should be a very nice middle-ground between fully edge or fully single-regional.

I haven’t used Fly.io a lot but I do like its approach and based on the feedback from others I definitely plan to give it a go with one of my side projects.

Fun fact: While writing this post, Fly.io announced 3GB Postgres or persisted volumes in the free tier 🥳

Conclusion

Serverless Platforms 2022

The diagram above summarises the platforms we discussed. Each one has its benefits and drawbacks, but the choice we have nowadays is amazing! I am pretty sure things will continue to evolve, improve, and we will see a lot more innovation in this space.

I have my favourites, you have yours, so let’s just keep on building 😉

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Kotlin http4k (via GraalVM Native Image) and Golang]]> https://www.lambrospetrou.com/articles/kotlin-http4k-graalvm-native-and-golang/ kotlin-http4k-graalvm-native-and-golang Sun, 26 Sep 2021 00:00:00 GMT First things first, this article is not a scientific performance comparison between Kotlin and Go (aka Golang). I want to show that someone can use lightweight frameworks with Kotlin to achieve small enough binaries that execute fast, and quite comparable to Go’s executables. One of my favourite deployment platforms is AWS Lambda, and it really makes a difference having small artifacts to upload, with very fast startup times, otherwise cold start invocations will essentially push latencies to the roof.

As part of a new project, I am trying to decide what stack to use as the backend. Let’s see how Go and Kotlin compare based on my very subjective and opinionated experience.

Go (favourite backend choice)

  • PRO: Great developer experience (i.e. tooling)
  • PRO: Great standard library
  • PRO: Small binary executables
  • PRO: Amazing performance (especially the low memory usage!)
  • CON: Some things are not available in quality libraries, hence might need boilerplate, or complete development (e.g. Machine Learning)

Kotlin (favourite language)

  • PRO: Amazingly expressive language
  • PRO: Great standard library & third-party libraries for anything
  • PRO: Good developer experience (at least when Gradle et al. work as expected)
  • PRO: Good overall performance
  • CON: Very high memory usage due to Java Virtual Machine (JVM)

The rest of the article will be a walkthrough for implementing a small API in Go and Kotlin, with a comparison of their performance against 200 simultaneous connections of 10s continuous traffic.

API

A very simple API with two endpoints, and SQLite as the database.

  • /?name=<requested-name>
    • Checks the database to see if <requested-name> exists in the users table and responds accordingly.
  • /add?name=<new-name>
    • Tries to add a new entry in the users table with name=<new-name> (without checking if it exists) and responds accordingly.

The table schema:

sqlite> .schema
CREATE TABLE users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT,
    UNIQUE(name)
);

Go

I will not paste here the full code for Go since it’s quite a few lines, and it’s not the main purpose of this article.

Check out the full source code at https://github.com/lambrospetrou/code-playground/blob/master/go-vs-kotlin-datastores/golang/main.go.

The only third-party dependency I use is crawshaw.io/sqlite library by David Crawshaw as the SQLite driver, who also wrote an amazing article about it.

Stats

Some interestings facts of the Go implementation.

  • Binary size: 9.7MB
$ ls -hl v1
-rwxr-xr-x  1 lambros  lambros   9.7M 26 Sep 20:10 v1
  • RSS memory at startup: 6.6MB
$ ps -eo pid,rss,%mem,%cpu,command | grep -e "./v1"
90176   6660  0.0   0.0 ./v1
  • Replay GET /add?name=i_am_ironman for 10s with 2 threads of 100 connections each.
    • Average requests per second: 15.11k (total: 303834)
    • RSS memory after replay: 17.9MB (!)
$ wrk -t2 -c100 -d10s --latency --timeout 1s http://localhost:8080/add\?name\=i_am_ironman
Running 10s test @ http://localhost:8080/add?name=i_am_ironman
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.89ms   15.52ms 675.26ms   97.49%
    Req/Sec    15.11k     2.10k   18.45k    58.91%
  Latency Distribution
     50%    3.00ms
     75%    3.62ms
     90%    4.51ms
     99%   51.22ms
  303834 requests in 10.10s, 52.45MB read
  Non-2xx or 3xx responses: 303833
Requests/sec:  30075.57
Transfer/sec:      5.19MB
  • Replay GET /?name=i_am_ironman for 10s with 2 threads of 100 connections each.
    • Average requests per second: 18.58k (total: 373386)
    • RSS memory after replay: 18.2MB (!)
$ wrk -t2 -c100 -d10s --latency --timeout 1s http://localhost:8080/\?name\=i_am_ironman
Running 10s test @ http://localhost:8080/?name=i_am_ironman
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.65ms  702.95us   9.48ms   73.62%
    Req/Sec    18.58k     2.45k   26.61k    74.26%
  Latency Distribution
     50%    2.60ms
     75%    2.99ms
     90%    3.50ms
     99%    4.75ms
  373386 requests in 10.10s, 53.06MB read
Requests/sec:  36960.38
Transfer/sec:      5.25MB

Small binary, fast startup, and extremely low memory usage 🚀

Kotlin

I am a huge fan of the http4k framework for building APIs with Kotlin. It is very small, modular, extremely versatile, and due to its philosophy “Server as a Function” it’s just an absolute joy to work with. It works both locally, and in AWS Lambda, and runs inside the JVM but also supports compiling down to native binary through GraalVM Native Image.

As a driver for SQLite I used SQLDelight. This is the first time I used it, and I am very pleasantly surprised by how nice it is. Definitely using it for all my SQL needs in the JVM. I tried to use JetBrains Exposed before SQLDelight but faced issues during the native image compilation, and its custom modelling is a no-go for me.

Check out the full source code at https://github.com/lambrospetrou/code-playground/tree/master/go-vs-kotlin-datastores/kotlin-sqlite.

As I mentioned above, I really love Kotlin “the language”, and in combination with http4k, it enables very lean servers. This is the entire server code.

package com.example

import com.squareup.sqldelight.db.SqlDriver
import com.squareup.sqldelight.sqlite.driver.JdbcSqliteDriver
import org.http4k.core.HttpHandler
import org.http4k.core.Method.GET
import org.http4k.core.Response
import org.http4k.core.Status
import org.http4k.core.Status.Companion.BAD_REQUEST
import org.http4k.core.Status.Companion.NOT_FOUND
import org.http4k.core.Status.Companion.OK
import org.http4k.routing.bind
import org.http4k.routing.routes
import org.http4k.server.ApacheServer
import org.http4k.server.asServer
import org.sqlite.SQLiteConfig

fun makeApp(db: Database): HttpHandler = routes(
    "/ping" bind GET to {
        Response(OK).body("pong")
    },

    "/add" bind GET to {
        val name = (it.query("name") ?: "").trim()
        when {
            name.isEmpty() ->
                Response(BAD_REQUEST).body("Invalid 'name' given: $name")
            else -> {
                val userQueries = db.usersQueries
                try {
                    userQueries.transaction {
                        userQueries.addUser(name = name)
                    }
                    Response(OK).body("Welcome to our community $name!")
                } catch (e: Exception) {
                    Response(Status.INTERNAL_SERVER_ERROR).body("Sadly we failed to register you: $name")
                }
            }
        }
    },

    "/" bind GET to {
        val name = (it.query("name") ?: "").trim()
        when {
            name.isEmpty() ->
                Response(BAD_REQUEST).body("Invalid 'name' given: $name")
            else -> {
                val userQueries = db.usersQueries
                userQueries.selectByName(name = name).executeAsOneOrNull()?.let {
                    Response(OK).body("Boom! We found you: $name")
                } ?: Response(NOT_FOUND).body("Sadly we could not find you: $name")
            }
        }
    }
)

fun main() {
    val port = (System.getenv("PORT") ?: "9000").toInt()

    val config = SQLiteConfig().apply {
        setSharedCache(true)
        setJournalMode(SQLiteConfig.JournalMode.WAL)
    }
    val driver: SqlDriver = JdbcSqliteDriver("jdbc:sqlite:/tmp/users.sqlite3", properties = config.toProperties())
    Database.Schema.create(driver)

    val app: HttpHandler = makeApp(db = Database(driver))
    val server = app.asServer(ApacheServer(port)).start()

    println(Runtime.getRuntime().availableProcessors())
    println("Server started on " + server.port())
}

Stats - JVM

In this section I am going to show the stats for running the above Kotlin code inside the JVM, the standard way. The following stats are without any custom JVM argument, just java -jar build/libs/HelloWorld.jar.

For context, I use the JVM provided by GraalVM:

$ java -version
openjdk version "16.0.2" 2021-07-20
OpenJDK Runtime Environment GraalVM CE 21.2.0 (build 16.0.2+7-jvmci-21.2-b08)
OpenJDK 64-Bit Server VM GraalVM CE 21.2.0 (build 16.0.2+7-jvmci-21.2-b08, mixed mode, sharing)
  • Fat-JAR size: 15MB
$ ls -hl build/libs/HelloWorld.jar
-rw-r--r--  1 lambros  lambros    15M 26 Sep 21:42 build/libs/HelloWorld.jar
  • RSS memory at startup: 143.9MB
$ ps -eo pid,rss,%mem,%cpu,command | grep -e "HelloWorld.jar"
 3460 143924  0.4   0.0 java -jar build/libs/HelloWorld.jar
  • Replay GET /add?name=i_am_ironman for 10s with 2 threads of 100 connections each.
    • Average requests per second: 7.17k (total: 142653)
    • RSS memory after replay: 1400MB (1.4GB)
$ wrk -t2 -c100 -d10s --latency --timeout 1s http://localhost:9000/add\?name\=i_am_ironman
Running 10s test @ http://localhost:9000/add?name=i_am_ironman
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    68.57ms  154.96ms 950.92ms   88.32%
    Req/Sec     7.17k     2.18k   12.00k    65.50%
  Latency Distribution
     50%  138.00us
     75%   39.81ms
     90%  269.81ms
     99%  722.75ms
  142653 requests in 10.01s, 27.75MB read
  Socket errors: connect 0, read 0, write 0, timeout 278
  Non-2xx or 3xx responses: 142653
Requests/sec:  14253.41
Transfer/sec:      2.77MB
  • Replay GET /?name=i_am_ironman for 10s with 2 threads of 100 connections each.
    • Average requests per second: 17.37k (total: 345639)
    • RSS memory after replay: 2500MB (2.5GB)
$ wrk -t2 -c100 -d10s --latency --timeout 1s http://localhost:9000/\?name\=i_am_ironman
Running 10s test @ http://localhost:9000/?name=i_am_ironman
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.41ms    4.36ms 139.93ms   93.48%
    Req/Sec    17.37k     3.04k   20.22k    92.00%
  Latency Distribution
     50%    2.68ms
     75%    4.68ms
     90%    6.92ms
     99%   15.54ms
  345639 requests in 10.01s, 1.01GB read
  Non-2xx or 3xx responses: 340797
Requests/sec:  34526.79
Transfer/sec:    103.35MB

So much memory used…

I also tried restricting the heap memory to 256MB (with java -Xmx256M -jar build/libs/HelloWorld.jar) but the results were roughly the same, even though the memory heap part was capped. The ApacheServer used here seems to be using lots of memory due to the many connections.

Stats - GraalVM Native Image

This section shows the boost we can get by compiling our code down to a native binary using GraalVM Native Image. This is a great piece of technology and hopefully it will keep evolving and improving.

This is the command I use to compile the fat-JAR down to a binary:

$ native-image --no-fallback -H:+ReportExceptionStackTraces --enable-url-protocols=https -jar build/libs/HelloWorld.jar build/libs/HelloWorld-native
  • Binary size: 46MB
  • RSS memory at startup: 19.7MB (!)
  • Replay GET /add?name=i_am_ironman for 10s with 2 threads of 100 connections each.
    • Average requests per second: 6.50k (total: 129385)
    • RSS memory after replay: 721.1MB
  • Replay GET /?name=i_am_ironman for 10s with 2 threads of 100 connections each.
    • Average requests per second: 17.17k (total: 341736)
    • RSS memory after replay: 1700MB (1.7GB)

As we can see, the memory usage is reduced significantly compared to the JVM runs, especially at the start, without introducing significant performance regression. Also, even though the binary size is larger than the fat-JAR, this is a standalone binary we can copy to any system and just run. In the fat-JAR case we still need to have the Java JVM installed in the system running the code. Although, to be fair, with the fat-JAR being less than 20MB cold starts in AWS Lambda should be non-existent.

Binary compression with UPX

UPX is a free, portable, extendable, high-performance executable packer for several executable formats.

Running upx -7 -k build/libs/HelloWorld-native will essentially take the GraalVM binary and compress it down significantly. The first time the executable runs, it decompresses the content and then proceeds to execution.

The size of the binary went from 46MB down to 18.9MB, without affecting the startup time in any meanigful way.

Stats - Summary

Artifact Size Initial RSS memory Final RSS memory Replay 1 Total Requests Replay 2 Total Requests
Go 9.7MB 6.6MB 18.2MB 303834 373386
JVM Fat-JAR 15MB 143.9MB 2500MB 142653 345639
GraalVM Native Image 46MB 19.7MB 1700MB 129385 341736
UPX 18.9MB - - - -

Conclusion

Please don’t get too scientific on me about the results. I know (and acknowledge) there are many nuances that affect the differences in performance, especially with SQLite parameters used, connection pooling in Go but not in Kotlin, etc.

The throughput & latency of all server versions examined above is satisfactory to me, so that’s not a concern.

The huge memory usage difference though is what still concerns me when using the JVM, even when the code gets compiled down to an executable binary. I expected that with GraalVM Native Image the memory use would be reduced a lot more, but I guess there is still lots of space for improvement.

I really like Kotlin the language though, so I am looking forward to the moment when memory won’t be an issue anymore. For the time being, I think I will stick with Go for long-running servers that will have thousands of parallel open connections. On the other hand, for AWS Lambda deployments Kotlin has become a very viable solution! Each Lambda invocation serves exactly one connection at a time, which means we only care about the binary size, and overall performance, and both of these are in the OK-territory (albeit the extra memory usage).

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Makefiles for execution coordination]]> https://www.lambrospetrou.com/articles/makefiles-for-execution-coordination/ makefiles-for-execution-coordination Sat, 25 Sep 2021 00:00:00 GMT As the title implies, this article is about the now ancient tool make [1] [2]. This little gem of software is used in our industry for at least 45 years now.

To be fair, I haven’t personally used it in big projects at work, but I regularly use it for my personal side projects since it gets the job done, and it provides a language agnostic common interface which means I can use it for Javascript projects, Kotlin projects, Golang projects, and others.

Enough with the intro though, in this article I will describe a very simple problem I wanted to solve recently and the good old make turned out to be the best solution, versus all the newer, shinier, and way more bloated modern alternatives.

Problem

The problem at hand is simple, and common to many people. I want a simple workflow tool to run some local programs in a specific order, each one potentially generating some output files to be used as input by subsequent programs in the workflow, and ideally parallel execution for the steps that do not depend on each other. A simple file processing data pipeline, to fully run locally on my laptop.

Simple eh… 🤪 😅

I did spend a few days looking for available tools but I was surprised that most of the modern workflow tooling is extremely bloated, complicated, and most of them made for “web scale distributed systems”, even though most people don’t really need that.

Candidate solutions:

  • https://snakemake.github.io
    • This is the best available tool I found, and if you notice its name it hints to the make tool. It has simple declarative way of expressing the workflow steps, their inputs and outputs, and the tool figures out the order in which to execute them. I really liked its documentation but I ended up not using it since it is built ontop of Python, and I don’t like Python…
  • Do-it-myself
    • The first thought I actually had was to just write a quick script in Go and shell out to the programs I wanted to execute, with as much parallelism and dynamism I needed. For a one-time script this is my goto, but I wanted to find a way to define the workflow declaratively, and learn something that could be used in the future for other more complicated flows as well.

make

I chatted with a friend about this and he instantly said “just use make”. I knew make was the main build tool for C/C++ projects, with fancy dynamic rules, running only the steps that have to run in the right order, and with good performance. I never used it though solely as an orchestration tool, with nothing to compile…

If you are not familiar with make, these are great resources on writing Makefiles:

Flow 1 - Parallel File Processing

The first example workflow does the following:

  1. Process all JSON files inside a directory data/. For each JSON file F we want to call some command C that will process F. This ideally should be done in parallel since there could be lots of files.
  2. Once all the files are processed, run another command which can continue the workflow execution.

Even though the workflow is simple, it still showcases how to use all the things we need:

  • Parallel execution of independent steps
  • Ordered execution of dependent steps

Makefile parallel files

The following Makefile (filename flow1.mk) implements the above workflow.

MAKEFILE_NAME = flow1.mk
DATAFILES = $(wildcard data/*.json)

default: boom

# Trigger a recursive `make` for the target `datafiles_all`.
# The critical argument is `--always-make` which will force the run all the time, 
# otherwise `make` will not do anything since the data files are not modified!
boom:
    $(MAKE) datafiles_all --always-make -f $(MAKEFILE_NAME)

# A trampoline target that depends on all the data files to force their processing.
datafiles_all: $(DATAFILES)
    @echo :: 'datafiles_all' finished!

# The target that corresponds to each JSON file in the `data/` directory.
data/%.json:
    @echo "processing single file:" $@
    @cat $@

If we run the above makefile using make -f flow1.mk we get the following output:

$ make -f flow1.mk
processing single file:  data/a.json
{"a": 1}
processing single file:  data/b.json
{"b": 2}
processing single file:  data/c.json
"c"
:: datafiles_all finished!

Pretty clear output showing that all three files in the data/ directory were processed.

Parallelism

If we run the above makefile with the additional -j 3 arguments, then all three files will be processed in parallel. This is not clear with the above example, so let’s make it a bit more complicated to showcase this as well.

# Run with `make -f flow1-parallel.mk -j 3` for parallelism 3
# or with `make -f flow1-parallel.mk -j $(nproc)` to use all processors.
# The default is to process each target one after the other.

MAKEFILE_NAME = flow1-parallel.mk
DATAFILES = $(wildcard data/*.json)

default: boom

# Trigger a recursive `make` for the target `datafiles_all`.
# The critical argument is `--always-make` which will force the run all the time, 
# otherwise `make` will not do anything since the data files are not modified!
boom:
    $(MAKE) datafiles_all --always-make -f $(MAKEFILE_NAME)

# A trampoline target that depends on all the data files to force their processing.
datafiles_all: $(DATAFILES)
    @echo :: 'datafiles_all' finished!

# Special override target for this specific file to simulate a long/slow execution.
data/a.json:
    @echo "processing slow file:" $@
    @sleep 2
    @echo still processing $@ ...
    @sleep 2
    @echo finished processing $@ ...

# The target that corresponds to each JSON file in the `data/` directory.
data/%.json:
    @echo "processing single file:" $@
    @cat $@

The only difference in this makefile is the newly added explicit target rule for data/a.json. By explicitly adding that rule, make will execute those commands instead of the generic statements defined by the data/%.json rule.

Let’s see how this runs with the default make invocation:

$ make -f flow1-parallel.mk
processing slow file: data/a.json
still processing data/a.json ...
finished processing data/a.json ...
processing single file: data/b.json
{"b": 2}
processing single file: data/c.json
"c"
:: datafiles_all finished!

As we can see, we need to completely finish the slow processing of data/a.json before proceeding with the rest files. Now let’s run it with parallelism 2:

$ make -f flow1-parallel.mk -j 2
processing slow file: data/a.json
processing single file: data/b.json
{"b": 2}
processing single file: data/c.json
"c"
still processing data/a.json ...
finished processing data/a.json ...
:: datafiles_all finished!

In this case, we can see that processing for data/a.json started as before, but before we even get to see the still processing data/a.json ... printout, the other two files are already processed, and then finally data/a.json completes.

make is smart enough with parallelism and no matter how many files, or target rules, we have in the Makefile it will respect the parallelism we specify with the -j N argument and execute the target steps accordingly without exceeding the specified parallelism.

Flow 2 - DAG

Another simple flow I want to show is how to make a pipeline of processes where some depend on each other, and some use local files/pipes as their communication mechanism.

Makefiles - DAG execution

What the diagram above means in plain English?

  1. Initially we start with the t1 and t2 targets executing in parallel.
  2. t1 writes an output file f1.txt and then proceeds to execute t3 which will read the f1.txt file as input (t3 depends on t6 though, so cannot start until t6 finishes as well).
  3. In parallel to t1/t3‘s execution flow, once t2 completes it will trigger t6, but t6 depends on t4 and t5 targets to have been completed first.
  4. t4 and t5 execute in parallel, and the output of t4 is being read by t5 through the file pipe /tmp/comm.fifo. Note that this is blocking communication, meaning that in order for t4 to manage to finish, t5 must also run in parallel to consume the content produced by t4.
  5. Once t6 finishes, t3 can execute.
  6. Once both t6 and t3 are complete, t7 will execute and complete our workflow.

The above workflow is modelled by the following makefile (flow2-dag.mk). Note how each target defines the dependencies it needs in order to properly coordinate the execution.

.PHONY: t1 t2 t3 t4 t5 t6 t7
default: t7

t1:
    @echo "t1"
    @echo "t1-content-output" > f1.txt
    @echo "t1 output file written!"

t2:
    @echo "t2"

t3: t1 t6
    @echo "t3"
    @cat f1.txt
    @echo "t1 output file printed!"

t4_5_setup:
    @rm -f /tmp/comm.fifo
    @mkfifo /tmp/comm.fifo

t4: t4_5_setup
    @echo "t4"
    @cat /usr/share/dict/words > /tmp/comm.fifo

t5: t4_5_setup
    @echo "t5"
    @echo "Total lines: " && wc -l < /tmp/comm.fifo

t6: t2 t4 t5
    @echo "t6"

t7: t3 t6
    @echo "t7"

Let’s run it and see if it works.

$ make -f flow2-dag.mk -j 4
t1
t4
t2
t5
Total lines:
t1 output file written!
  235886
t6
t3
t1-content-output
t1 output file printed!
t7

Boom 🥳 It works as expected 🚀 (exercise to the reader to confirm…)

Note: This workflow requires parallelism of at least 2, otherwise targets t4 and t5 will deadlock. Target t4 will be the only one running, and once it fills up the file pipe it will block until someone consumes it. But since t5 is not running the workflow will be stuck. In scenarios like this, either always make sure to run this with some parallelism, or use normal files as communication, as we did between t1 and t3.

Conclusion

There is no doubt that make is amazingly powerful and flexible enough to achieve any kind of workflow execution. I barely even covered its functionality. I expected it to be much more cumbersome to use due to the complexity I see in how most of the C/C++ projects are using it.

I was pleasantly surprised! For these use-cases when I don’t want complicated bloated tools, make fits the bill perfectly! The fact that make is available on almost any system, with super fast execution, makes it a great tool in my toolkit.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Keeping the brain engaged]]> https://www.lambrospetrou.com/articles/keeping-the-brain-engaged/ keeping-the-brain-engaged Sun, 12 Sep 2021 00:00:00 GMT During the past few weeks I have been feeling that I am not as engaged as I want to be at work, and neither at my outside of work activites. I decided to do some kind of retrospective, going over the past few years to remember what kept my brain excited and engaged. Not really to my surprise, but I realised that it’s not always about projects at work, nor hobbies, nor friends; it could be any of them.

When at times I was working on exciting projects at work, I left the office but my brain was still crunching, even subconciously. While riding the London Underground home I might have been evaluating different solutions to the problem at hand, or thinking about new features to improve the product, or trying to think through what went wrong in that solution that should work but didn’t. It wasn’t rare that I would wake up suddenly in the middle of the night with a brain flash about a software bug I had overlooked, even ones I wasn’t even aware they existed before going to bed, along with its fix which was quite obvious at that moment.

This happens a few times with my hobbies as well, like when I play volleyball. I get flashes later of what I should do better, things I overlooked on the opponent’s side, or how the game situation demanded a certain play but I played differently. Discussions with friends are also a good trigger for these unconcious sudden revelations, especially when we discuss ideas and projects about potential startups. We start from a simple pitch, and end up with business plans and features that need years to implement — talk about a long-term vision. Or when I am evaluating different companies or funds to invest in the stock market, and trying to think of how they would evolve in the future, areas they would expand, potential fall-throughs. And in all these cases the majority of the thinking is done after the fact, not while it’s taking place, but at times when my brain is able to drift freely.

Coincidentally, last week I stumbled upon the following essays by Paul Graham that really stroke a cord and resonated immediately with all the above.

What Doesn’t Seem Like Work

Starts talking about how his father loved solving math problems, how he never got bored of them, and concludes with:

It seemed curious that the same task could be painful to one person and pleasant to another, […]. I didn’t realize how hard it can be to decide what you should work on, and that you sometimes have to figure it out from subtle clues, like a detective solving a case in a mystery novel. So I bet it would help a lot of people to ask themselves about this explicitly. What seems like work to other people that doesn’t seem like work to you?

The Top Idea in Your Mind

The first paragraph nicely summarizes what I described in the intro of this article:

I realized recently that what one thinks about in the shower in the morning is more important than I’d thought. I knew it was a good time to have ideas. Now I’d go further: now I’d say it’s hard to do a really good job on anything you don’t think about in the shower.

Then at some point he mentions the following, referring to the unforced thinking our brain does unconciously.

I think most people have one top idea in their mind at any given time. That’s the idea their thoughts will drift toward when they’re allowed to drift freely. And this idea will thus tend to get all the benefit of that type of thinking, while others are starved of it. Which means it’s a disaster to let the wrong idea become the top one in your mind.

And concludes with (emphasis is mine):

I suspect a lot of people aren’t sure what’s the top idea in their mind at any given time. I’m often mistaken about it. I tend to think it’s the idea I’d want to be the top one, rather than the one that is. But it’s easy to figure this out: just take a shower. What topic do your thoughts keep returning to? If it’s not what you want to be thinking about, you may want to change something.

A Projects of One’s Own

This is an awesome article and I felt what Paul describes here a lot during the past few months, missing this excitement and satisfaction of working on projects of mine. Projects where you are in control, you do them because you like and want to, without useless and unnecessary processes or people slowing you down.

Putting some excerpts below that I find very interesting.

Working on a project of your own is as different from ordinary work as skating is from walking. It’s more fun, but also much more productive.

There is something special about working on a project of your own. I wouldn’t say exactly that you’re happier. A better word would be excited, or engaged.

You feel as if you’re an animal in its natural habitat, doing what you were meant to do — not always happy, maybe, but awake and alive.

People who’ve never experienced the thrill of working on a project they’re excited about can’t distinguish this kind of working long hours from the kind that happens in sweatshops and boiler rooms, but they’re at opposite ends of the spectrum. That’s why it’s a mistake to insist dogmatically on “work/life balance.” Indeed, the mere expression “work/life” embodies a mistake: it assumes work and life are distinct. For those to whom the word “work” automatically implies the dutiful plodding kind, they are. But for the skaters, the relationship between work and life would be better represented by a dash than a slash. I wouldn’t want to work on anything I didn’t want to take over my life.

If you can find the right people, you only have to tell them what to do at the highest level. They’ll handle the details. Indeed, they insist on it. For a project to feel like your own, you must have sufficient autonomy. You can’t be working to order, or slowed down by bureaucracy.

[…] Ideally we can have the best of both worlds: to be deliberate in choosing to work on projects of our own, and carelessly confident in starting new ones.

Conclusion

My take-away from the above is very simple. I need, and want, to maximise the time that my brain is engaged in things I like, things I get excited about, and things I get satisfaction from. At work I need to chase certain projects and avoid others. With hobbies I need to drop the ones that are just time-wasting and focus on the ones that bring excitement. And with side projects, well, I need to restart doing!

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Bits of Unsolicited Advice by Kevin Kelly]]> https://www.lambrospetrou.com/articles/bits-of-unsolicited-advice-kevin-kelly/ bits-of-unsolicited-advice-kevin-kelly Mon, 26 Apr 2021 00:00:00 GMT Today, I stumbled upon two articles with some amazing pieces of advice from Kevin Kelly.

I loved them so much I wanted to share them here as well. However, all bragging rights and attribution should be given to the original articles:

I also added these to my Read-Watch-Listen list.

68 Bits of Unsolicited Advice

  • Learn how to learn from those you disagree with, or even offend you. See if you can find the truth in what they believe.
  • Being enthusiastic is worth 25 IQ points.
  • Always demand a deadline. A deadline weeds out the extraneous and the ordinary. It prevents you from trying to make it perfect, so you have to make it different. Different is better.
  • Don’t be afraid to ask a question that may sound stupid because 99% of the time everyone else is thinking of the same question and is too embarrassed to ask it.
  • Being able to listen well is a superpower. While listening to someone you love keep asking them “Is there more?”, until there is no more.
  • A worthy goal for a year is to learn enough about a subject so that you can’t believe how ignorant you were a year earlier.
  • Gratitude will unlock all other virtues and is something you can get better at.
  • Treating a person to a meal never fails, and is so easy to do. It’s powerful with old friends and a great way to make new friends.
  • Don’t trust all-purpose glue.
  • Reading to your children regularly will bond you together and kickstart their imaginations.
  • Never use a credit card for credit. The only kind of credit, or debt, that is acceptable is debt to acquire something whose exchange value is extremely likely to increase, like in a home. The exchange value of most things diminishes or vanishes the moment you purchase them. Don’t be in debt to losers.
  • Pros are just amateurs who know how to gracefully recover from their mistakes.
  • Extraordinary claims should require extraordinary evidence to be believed.
  • Don’t be the smartest person in the room. Hangout with, and learn from, people smarter than yourself. Even better, find smart people who will disagree with you.
  • Rule of 3 in conversation. To get to the real reason, ask a person to go deeper than what they just said. Then again, and once more. The third time’s answer is close to the truth.
  • Don’t be the best. Be the only.
  • Everyone is shy. Other people are waiting for you to introduce yourself to them, they are waiting for you to send them an email, they are waiting for you to ask them on a date. Go ahead.
  • Don’t take it personally when someone turns you down. Assume they are like you: busy, occupied, distracted. Try again later. It’s amazing how often a second try works.
  • The purpose of a habit is to remove that action from self-negotiation. You no longer expend energy deciding whether to do it. You just do it. Good habits can range from telling the truth, to flossing.
  • Promptness is a sign of respect.
  • When you are young spend at least 6 months to one year living as poor as you can, owning as little as you possibly can, eating beans and rice in a tiny room or tent, to experience what your “worst” lifestyle might be. That way any time you have to risk something in the future you won’t be afraid of the worst case scenario.
  • Trust me: There is no “them”.
  • The more you are interested in others, the more interesting they find you. To be interesting, be interested.
  • Optimize your generosity. No one on their deathbed has ever regretted giving too much away.
  • To make something good, just do it. To make something great, just re-do it, re-do it, re-do it. The secret to making fine things is in remaking them.
  • The Golden Rule will never fail you. It is the foundation of all other virtues.
  • If you are looking for something in your house, and you finally find it, when you’re done with it, don’t put it back where you found it. Put it back where you first looked for it.
  • Saving money and investing money are both good habits. Small amounts of money invested regularly for many decades without deliberation is one path to wealth.
  • To make mistakes is human. To own your mistakes is divine. Nothing elevates a person higher than quickly admitting and taking personal responsibility for the mistakes you make and then fixing them fairly. If you mess up, fess up. It’s astounding how powerful this ownership is.
  • Never get involved in a land war in Asia.
  • You can obsess about serving your customers/audience/clients, or you can obsess about beating the competition. Both work, but of the two, obsessing about your customers will take you further.
  • Show up. Keep showing up. Somebody successful said: 99% of success is just showing up.
  • Separate the processes of creation from improving. You can’t write and edit, or sculpt and polish, or make and analyze at the same time. If you do, the editor stops the creator. While you invent, don’t select. While you sketch, don’t inspect. While you write the first draft, don’t reflect. At the start, the creator mind must be unleashed from judgement.
  • If you are not falling down occasionally, you are just coasting.
  • Perhaps the most counter-intuitive truth of the universe is that the more you give to others, the more you’ll get. Understanding this is the beginning of wisdom.
  • Friends are better than money. Almost anything money can do, friends can do better. In so many ways a friend with a boat is better than owning a boat.
  • This is true: It’s hard to cheat an honest man.
  • When an object is lost, 95% of the time it is hiding within arm’s reach of where it was last seen. Search in all possible locations in that radius and you’ll find it.
  • You are what you do. Not what you say, not what you believe, not how you vote, but what you spend your time on.
  • If you lose or forget to bring a cable, adapter or charger, check with your hotel. Most hotels now have a drawer full of cables, adapters and chargers others have left behind, and probably have the one you are missing. You can often claim it after borrowing it.
  • Hatred is a curse that does not affect the hated. It only poisons the hater. Release a grudge as if it was a poison.
  • There is no limit on better. Talent is distributed unfairly, but there is no limit on how much we can improve what we start with.
  • Be prepared: When you are 90% done any large project (a house, a film, an event, an app) the rest of the myriad details will take a second 90% to complete.
  • When you die you take absolutely nothing with you except your reputation.
  • Before you are old, attend as many funerals as you can bear, and listen. Nobody talks about the departed’s achievements. The only thing people will remember is what kind of person you were while you were achieving.
  • For every dollar you spend purchasing something substantial, expect to pay a dollar in repairs, maintenance, or disposal by the end of its life.

-Anything real begins with the fiction of what could be. Imagination is therefore the most potent force in the universe, and a skill you can get better at. It’s the one skill in life that benefits from ignoring what everyone else knows.

  • When crisis and disaster strike, don’t waste them. No problems, no progress.
  • On vacation go to the most remote place on your itinerary first, bypassing the cities. You’ll maximize the shock of otherness in the remote, and then later you’ll welcome the familiar comforts of a city on the way back.
  • When you get an invitation to do something in the future, ask yourself: would you accept this if it was scheduled for tomorrow? Not too many promises will pass that immediacy filter.
  • Don’t say anything about someone in email you would not be comfortable saying to them directly, because eventually they will read it.
  • If you desperately need a job, you are just another problem for a boss; if you can solve many of the problems the boss has right now, you are hired. To be hired, think like your boss.
  • Art is in what you leave out.
  • Acquiring things will rarely bring you deep satisfaction. But acquiring experiences will.
  • Rule of 7 in research. You can find out anything if you are willing to go seven levels. If the first source you ask doesn’t know, ask them who you should ask next, and so on down the line. If you are willing to go to the 7th source, you’ll almost always get your answer.
  • How to apologize: Quickly, specifically, sincerely.
  • Don’t ever respond to a solicitation or a proposal on the phone. The urgency is a disguise.
  • When someone is nasty, rude, hateful, or mean with you, pretend they have a disease. That makes it easier to have empathy toward them which can soften the conflict.
  • Eliminating clutter makes room for your true treasures.
  • You really don’t want to be famous. Read the biography of any famous person.
  • Experience is overrated. When hiring, hire for aptitude, train for skills. Most really amazing or great things are done by people doing them for the first time.
  • A vacation + a disaster = an adventure.
  • Buying tools: Start by buying the absolute cheapest tools you can find. Upgrade the ones you use a lot. If you wind up using some tool for a job, buy the very best you can afford.
  • Learn how to take a 20-minute power nap without embarrassment.
  • Following your bliss is a recipe for paralysis if you don’t know what you are passionate about. A better motto for most youth is “master something, anything”. Through mastery of one thing, you can drift towards extensions of that mastery that bring you more joy, and eventually discover where your bliss is.
  • I’m positive that in 100 years much of what I take to be true today will be proved to be wrong, maybe even embarrassingly wrong, and I try really hard to identify what it is that I am wrong about today.
  • Over the long term, the future is decided by optimists. To be an optimist you don’t have to ignore all the many problems we create; you just have to imagine improving our capacity to solve problems.
  • The universe is conspiring behind your back to make you a success. This will be much easier to do if you embrace this pronoia.

99 Additional Bits of Unsolicited Advice

  • That thing that made you weird as a kid could you make great as an adult — if you don’t lose it.
  • If you have any doubt at all about being able to carry a load in one trip, do yourself a huge favor and make two trips.
  • What you get by achieving your goals is not as important as what you become by achieving your goals. At your funeral people will not recall what you did; they will only remember how you made them feel.
  • Recipe for success: under-promise and over-deliver.
  • It’s not an apology if it comes with an excuse. It is not a compliment if it comes with a request.
  • Jesus, Superman, and Mother Teresa never made art. Only imperfect beings can make art because art begins in what is broken.
  • If someone is trying to convince you it’s not a pyramid scheme, it’s a pyramid scheme.
  • Learn how to tie a bowline knot. Practice in the dark. With one hand. For the rest of your life you’ll use this knot more times than you would ever believe.
  • If something fails where you thought it would fail, that is not a failure.
  • Be governed not by the tyranny of the urgent but by the elevation of the important.
  • Leave a gate behind you the way you first found it.
  • The greatest rewards come from working on something that nobody has a name for. If you possibly can, work where there are no words for what you do.
  • A balcony or porch needs to be at least 6 feet (2m) deep or it won’t be used.
  • Don’t create things to make money; make money so you can create things. The reward for good work is more work.
  • In all things — except love — start with the exit strategy. Prepare for the ending. Almost anything is easier to get into than out of.
  • Train employees well enough they could get another job, but treat them well enough so they never want to.
  • Don’t aim to have others like you; aim to have them respect you.
  • The foundation of maturity: Just because it’s not your fault doesn’t mean it’s not your responsibility.
  • A multitude of bad ideas is necessary for one good idea.
  • Being wise means having more questions than answers.
  • Compliment people behind their back. It’ll come back to you.
  • Most overnight successes — in fact any significant successes — take at least 5 years. Budget your life accordingly.
  • You are only as young as the last time you changed your mind.
  • Assume anyone asking for your account information for any reason is guilty of scamming you, unless proven innocent. The way to prove innocence is to call them back, or login to your account using numbers or a website that you provide, not them. Don’t release any identifying information while they are contacting you via phone, message or email. You must control the channel.
  • Sustained outrage makes you stupid.
  • Be strict with yourself and forgiving of others. The reverse is hell for everyone.
  • Your best response to an insult is “You’re probably right.” Often they are.
  • The worst evils in history have always been committed by those who truly believed they were combating evil. Beware of combating evil.
  • If you can avoid seeking approval of others, your power is limitless.
  • When a child asks an endless string of “why?” questions, the smartest reply is, “I don’t know, what do you think?”
  • To be wealthy, accumulate all those things that money can’t buy.
  • Be the change you wish to see.
  • When brainstorming, improvising, jamming with others, you’ll go much further and deeper if you build upon each contribution with a playful “yes — and” example instead of a deflating “no — but” reply.
  • Work to become, not to acquire.
  • Don’t loan money to a friend unless you are ready to make it a gift.
  • On the way to a grand goal, celebrate the smallest victories as if each one were the final goal. No matter where it ends you are victorious.
  • Calm is contagious.
  • Even a foolish person can still be right about most things. Most conventional wisdom is true.
  • Always cut away from yourself.
  • Show me your calendar and I will tell you your priorities. Tell me who your friends are, and I’ll tell you where you’re going.
  • When hitchhiking, look like the person you want to pick you up.
  • Contemplating the weaknesses of others is easy; contemplating the weaknesses in yourself is hard, but it pays a much higher reward.
  • Worth repeating: measure twice, cut once.
  • Your passion in life should fit you exactly; but your purpose in life should exceed you. Work for something much larger than yourself.
  • If you can’t tell what you desperately need, it’s probably sleep.
  • When playing Monopoly, spend all you have to buy, barter, or trade for the Orange properties. Don’t bother with Utilities.
  • If you borrow something, try to return it in better shape than you received it. Clean it, sharpen it, fill it up.
  • Even in the tropics it gets colder at night than you think. Pack warmly.
  • To quiet a crowd or a drunk, just whisper.
  • Writing down one thing you are grateful for each day is the cheapest possible therapy ever.
  • When someone tells you something is wrong, they’re usually right. When someone tells you how to fix it, they’re usually wrong.
  • If you think you saw a mouse, you did. And, if there is one, there are more.
  • Money is overrated. Truly new things rarely need an abundance of money. If that was so, billionaires would have a monopoly on inventing new things, and they don’t. Instead almost all breakthroughs are made by those who lack money, because they are forced to rely on their passion, persistence and ingenuity to figure out new ways. Being poor is an advantage in innovation.
  • Ignore what others may be thinking of you, because they aren’t.
  • Avoid hitting the snooze button. That’s just training you to oversleep.
  • Always say less than necessary.
  • You are given the gift of life in order to discover what your gift in life is. You will complete your mission when you figure out what your mission is. This is not a paradox. This is the way.
  • Don’t treat people as bad as they are. Treat them as good as you are.
  • It is much easier to change how you think by changing your behavior, than it is to change your behavior by changing how you think. Act out the change you seek.
  • You can eat any dessert you want if you take only 3 bites.
  • Each time you reach out to people, bring them a blessing; then they’ll be happy to see you when you bring them a problem.
  • Bad things can happen fast, but almost all good things happen slowly.
  • Don’t worry how or where you begin. As long as you keep moving, your success will be far from where you start.
  • When you confront a stuck bolt or screw: righty tighty, lefty loosey.
  • If you meet a jerk, overlook them. If you meet jerks everywhere everyday, look deeper into yourself.
  • Dance with your hips.
  • We are not bodies that temporarily have souls. We are souls that temporarily have bodies.
  • You can reduce the annoyance of someone’s stupid belief by increasing your understanding of why they believe it.
  • If your goal does not have a schedule, it is a dream.
  • All the greatest gains in life — in wealth, relationships, or knowledge —come from the magic of compounding interest — amplifying small steady gains. All you need for abundance is to keep adding 1% more than you subtract on a regular basis.
  • The greatest breakthroughs are missed because they look like hard work.
  • People can’t remember more than 3 points from a speech.
  • I have never met a person I admired who did not read more books than I did.
  • The greatest teacher is called “doing”.
  • Finite games are played to win or lose. Infinite games are played to keep the game going. Seek out infinite games because they yield infinite rewards.
  • Everything is hard before it is easy. The day before something is a breakthrough, it’s a stupid idea.
  • A problem that can be solved with money is not really a problem.
  • When you are stuck, sleep on it. Let your subconscious work for you.
  • Your work will be endless, but your time is finite. You cannot limit the work so you must limit your time. Hours are the only thing you can manage.
  • To succeed, get other people to pay you; to become wealthy, help other people to succeed.
  • Children totally accept — and crave — family rules. “In our family we have a rule for X” is the only excuse a parent needs for setting a family policy. In fact, “I have a rule for X” is the only excuse you need for your own personal policies.
  • All guns are loaded.
  • Many backward steps are made by standing still.
  • This is the best time ever to make something. None of the greatest, coolest creations 20 years from now have been invented yet. You are not late.
  • No rain, no rainbow.
  • Every person you meet knows an amazing lot about something you know virtually nothing about. Your job is to discover what it is, and it won’t be obvious.
  • You don’t marry a person, you marry a family.
  • Always give credit, take blame.
  • Be frugal in all things, except in your passions splurge.
  • When making something, always get a few extras — extra material, extra parts, extra space, extra finishes. The extras serve as backups for mistakes, reduce stress, and fill your inventory for the future. They are the cheapest insurance.
  • Something does not need to be perfect to be wonderful. Especially weddings.
  • Don’t let your email inbox become your to-do list.
  • The best way to untangle a knotty tangle is not to “untie” the knots, but to keep pulling the loops apart wider and wider. Just make the mess as big, loose and open as possible. As you open up the knots they will unravel themselves. Works on cords, strings, hoses, yarns, or electronic cables.
  • Be a good ancestor. Do something a future generation will thank you for. A simple thing is to plant a tree.
  • To combat an adversary, become their friend.
  • Take one simple thing — almost anything — but take it extremely seriously, as if it was the only thing in the world, or maybe the entire world is in it — and by taking it seriously you’ll light up the sky.
  • History teaches us that in 100 years from now some of the assumptions you believed will turn out to be wrong. A good question to ask yourself today is “What might I be wrong about?”
  • Be nice to your children because they are going to choose your nursing home.
  • Advice like these are not laws. They are like hats. If one doesn’t fit, try another.
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Minibri Temp - Asciidoc support]]> https://www.lambrospetrou.com/articles/minibri-temp-asciidoc/ minibri-temp-asciidoc Wed, 17 Mar 2021 00:00:00 GMT Minibri Temp now supports files formatted in Asciidoc. The uploaded file needs to have .adoc or .asciidoc extension to go through Asciidoc conversion.

Check out an example of a rendered document at https://temp.minibri.com/view/asciidoc-guide.

The source file of 6000+ lines which generated the above output can be found at this gist.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[The Read-Watch-Listen list]]> https://www.lambrospetrou.com/articles/the-read-watch-listen-list/ the-read-watch-listen-list Sun, 14 Mar 2021 00:00:00 GMT I often share links to content I read, watch, or listen in my social media accounts (mostly Twitter). However, many times I am trying to find a link I know I shared before, but don’t remember exactly where it was hosted or who created it. And then I end up searching or scrolling my tweets in an attempt to find it. Looking for a needle in a haystack…

I decided to create a page where I will keep a list of content that I find interesting, and is worthy of sharing with others. This will be a curated list of things, and not just a dump of my tweets, with just what I consider great.

I spent a few hours scrolling through my tweets for the past three years and I extracted the content that I still find interesting, and it’s not behind a paywall at the moment. You can find the new page at https://www.lambrospetrou.com/read-watch-listen, and at the corresponding navigation link at the top of all pages on this website. The list is ordered with the content I most recently read at the top, therefore new items will always go at the top.

I also created a dedicated RSS feed separate from my articles RSS feed in order to provide an easy way for people to consume this list using their favourite feed reader.

I hope more people find the content of this list interesting. Feel free to get in touch if you have any feedback, or want to share great content with me.

And just for the history records, a screenshot of the list at the article’s time of writing 👇🏼

The Read-Watch-Listen list screenshot

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[A journey through email providers (Zoho Mail, Private Email, Gmail, HEY, Google Domains)]]> https://www.lambrospetrou.com/articles/a-journey-through-email-providers/ a-journey-through-email-providers Sun, 07 Mar 2021 00:00:00 GMT Context

I use several domains for my websites and side projects, and for some of them I want to have email addresses that use the correspoding domains, instead of advertising a Gmail address. For example, I want my personal domain to use the @lambrospetrou.com domain, instead of @gmail.com.

There are hundreds of email providers available, and over the past year I tried a couple of them, and read about tens more.

The process setting up the custom domain email is roughly the same across all providers:

  1. You signup and pay for their service.
  2. The service provider gives you their MX records to put in your domain’s DNS configuration.
  3. You update your domain’s DNS and boom, your custom domain email address is ready.

However, the email experience you get from each provider differs substantially…

Namecheap Private Email

Almost a year ago I decided to go with the cheapest option, whilst also having good ratings. For my personal domain (this website 👋) I used Namecheap Private Email since I was happy with Namecheap as a domain registrar a couple of times.

The offering cannot be simpler for around £9 ($12) per year 🤯:

  • 1 free mailbox included
  • 5GB for emails
  • 2GB for files

I only used Private Email’s website, and even though it did not dazzle me, it was also not missing something important. Apart from one thing…

The spam filter algorithm is horrible, entirely inexistent. I have a few articles about AWS Lambda, Amazon S3, and AWS in general, and since they got picked up by some spammers, I am getting tens of emails that can be considered poster emails for testing spam algorithms.

So, since the year is almost up, I decided to not renew again. Next one…

Zoho Mail

For a different domain, a side project, I used Zoho Mail. The Zoho Suite is sometimes referred to as the cheaper Google G-Suite, since it is a collection of services and email is just one of them.

Similar to the other domain, I used the cheapest plan offered, Zoho Mail Lite at £12 ($16):

  • 5GB total
  • Up to 250MB attachments
  • Lots of other stuff I don’t care…

I will be honest, this email is of very low use, and I didn’t publish it in any way so I cannot directly compare it with Private Email in terms of spam filters. However, in roughly 6 months I haven’t seen a spam email at all.

Zoho Mail screenshot

Once again, I only used the website to access this email. The email view is quite standard, but I found the rest UI too complicated, and I got the sense that Zoho tries too much to look like a toolbox rather than a focused tool. Especially when I was navigating through the settings in the first few days to setup everything I got lost more times than I would like to admit… I haven’t used any of the Calendar, Tasks, Notes, and other services, so cannot comment on those.

Overall, I have mixed feelings for Zoho Mail. It’s cheap, simple for just email, but I have to use it more often to see if the rest of the services in the package make up for the intrinsic complexity.

Google Workspace - G-Suite

Google Workspace is what is famously known as G-Suite until last year when Google did a rebranding. G-Suite is probably the most popular among professional email providers, due to Gmail’s popularity, and the rest of the Google services (Docs, Drive, Calendar, etc.). The only one close in my opinion is the Microsoft 365 family of services. I personally hate Microsoft’s plan breakdown, and I don’t find Outlook to be on par with Gmail.

Google Workspace - GSuite home page

I have considered subscribing to the Business Starter plan for Google Workspace for months now (still haven’t):

  • 30GB cloud storage per user
  • £4.60 (~$6.50) per month
  • Integration with all the Google services with some business oriented features (e.g. for Google Meet)

One thing that holds me back is that some of the business versions of the services lack certain features that are available in the corresponding consumer versions. Since I am mostly interested in Gmail, it’s not an actual issue to be honest, especially when compared with other email providers. But if I would be paying that monthly amount then I would like to use all the features I get with that. At times I would probably get into conflict, between my personal Google account and the business one, having to choose which one to use since I am a heavy Google services user.

Google Workspace (G-Suite) is my top choice so far for a dedicated email provider, and I am sure I will find a way to juggle my data between my personal and the business account.

HEY for You

HEY for You is a brand new email service provided by the great team at Basecamp. HEY’s price is almost double than Google Workspace’s at roughly £87 ($120 - $99 + 20% VAT) per year. The big promise from HEY justifying the high price is that it revolutionizes how you use email, taking some modern techniques cued from social media services (e.g. The Feed, the Just let it flow philosophy), and avoids adding “management features” in an effort to make email simple.

HEY for You home page

I read through the whole feature list and watched the YouTube videos presenting HEY, and it’s quite intriguing. Having said that, I think that one of the reasons I am even considering this product at such a steep price is just because of Basecamp, the team, the company. Last year I read three of the books they wrote (Getting Real, REWORK, Shape Up) and loved them, so I might be biased thinking I would like their email product too 😅

I am definitely going to at least signup for the trial and give this a go in the next few days. I haven’t yet pulled the trigger because custom domain support is still only available for Hey for Work, the business version of the service, which is even more expensive.

Google Domains

I stumbled upon this solution today, out of nowhere, while reading some Reddit comments.

Google Domains is Google’s new domain registrar service, still carrying the BETA status. However, today I was pleasantly surprised when I found out that it also provides a free email forwarding service.

As it turns out, you can have up to 100 aliases per registered domain, and each can be forwarded to a different email address, including personal Gmail accounts. I immediately transferred two of my domains to Google Domains and tried the email forwarding feature, and it actually works great 🥳

In addition, according to the docs, email forwarding can also work alongside G-Suite so you can have both enabled at the same time if needed for the same domain.

Email forwarding is obviously not an apples-to-apples comparison to all the services I previously mentioned, but since the original goal was custom domain email address, it fully satisfies my needs. I just forward my domain’s email addresses to my personal Gmail address and boom, everything ends up in the same account which is easy to manage, and with the ability to send and reply emails using the domain address it’s complete.

Conclusion

Google Domains solved my immediate problem in the best way possible since I don’t have to login and use any other mediocre email provider, I have an amazing spam filter, and it’s all within Gmail which I love.

Having said that, I am still in search for a dedicated offering for my non-personal domains used for some side projects, future mini-businesses. Google Workspaces and HEY are my top two finalists, and I guess I will have to finally tryout HEY before picking the winner.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Minibri Temp — The Deep Dive]]> https://www.lambrospetrou.com/articles/minibri-temp-the-deep-dive/ minibri-temp-the-deep-dive Mon, 01 Mar 2021 00:00:00 GMT Minibri Temp home page

Try Minibri Temp at https://temp.minibri.com.

Inception Story

Minibri Temp allows you to upload a file which contains either HTML code (including CSS and JavaScript) or content in one of the supported formats, e.g. Markdown, and soon Asciidoc. Once the file is uploaded, the content is converted to HTML, and you get a link to view the rendered HTML content. You also specify a duration, after which the URL will expire and the uploaded content will be deleted.

This service seems incredibly simple, because it is 😅 Markdown converters are very popular, but my plan is to add support for a few more source formats, which is what differentiates this from everything else since I cannot really find anything supporting non-Markdown content.

I had several moments last year that I wanted to share a link with a friend or a colleague to show them some content. To do that, I had to run some local command line tool on my laptop to convert it to HTML, then login to my AWS account (or Github) and upload the file to my website, and then finally share a link with them. Minibri Temp just makes this simple use-case a 1-step action! 🥳 Google Drive and Dropbox also make this easy but I don’t have them on all my machines, so they are not there when I need them.

I developed the application over two weekends, so it’s nothing record-breaking or something that’s going to win the Alan Turing award, but in this article I would like to do a deep dive into its architecture and some technical decisions. I want to show that using just a few core services allows someone to build useful applications, whilst also having fun, and at a cost of almost zero.

Architecture Overview

Minibri Temp High Level arch

As the diagram above shows the user’s browser interacts with three main systems.

  1. The static assets are loaded from Netlify Edge, which is Netlify’s smart Content Delivery Network (CDN).
  2. The uploaded content is sent to a Netlify Function, which is Netlify’s serverless compute platform built ontop of AWS Lambda. This function does some content validation, and creates one HTML file with the converted content.
  3. The created HTML file is immutable and is stored on Amazon S3, Amazon’s object storage offering on AWS. The page returned to the user directly fetches the converted HTML file from Amazon S3 (using presigned URLs).

The last step is a bit debateable and I could serve the converted HTML files through Netlify Edge as well, having the S3 bucket acting as an origin. I did not go with this approach for now to avoid having to make the files stored on S3 public, therefore all access to the files uses presigned URLs, which gives me control on who can access them and for how long.

No JavaScript required

It might be a bit unusual nowadays, but I want the website to be fully functional without JavaScript required in the user’s browser. Obviously, I will introduce some JavaScript in the future, e.g. for drag & drop the file instead of manual selection, but I want 100% of the functionality to work without JavaScript needed.

Content server inside an iframe

If you noticed in the architecture overview I mentioned that the output HTML is fetched using an S3 presigned URL. In addition, if you examine the actual page returned when viewing the generated content (e.g. this sample) you will see that it’s basically a very simple skeleton page that contains only an <iframe> with its src attribute set to the S3 presigned URL.

Let’s see why.

I have the following requirements:

  • The generated HTML for the converted content should be immutable. After its creation I should not need to ever touch it again, so that it can be served from S3 or any CDN service.
  • I want to be able to update the skeleton of the view page, for example to customize the header, or maybe in the future add a footer, etc. These changes are not related to the converted content, but the surrounding parts.

I thought about three possible solutions to satisfy the immutability requirement:

  1. Generate the HTML file for the converted content, and also include all the HTML needed for my own purposes (e.g. header). Each HTML file will be self-contained completely, and will be the only thing needed to be served to the user.
  2. Generate the HTML file for the converted content, and include an <iframe> at the header section as a placeholder which has its src attribute set to the URL serving the header.
  3. Generate the HTML file for the converted content and store it on its own in S3. The HTML served to the user would be a different HTML page that contains an <iframe> with its src attribute set to the HTML file in S3 containing the converted content.

Solution 1 is the easiest, and probably the one with the best caching since it can be cached literally forever. However, it does not allow for updates which is a deal-breaker (see second requirement). Solution 2 is OK since it allows me to update the header section anytime I want, and still benefits from immutable, cacheable HTML files. Solution 3 provides flexibility, and keeping the actual HTML file for the converted content separate from the skeleton page is good for future-proofing. Therefore, I went with solution 3.

Direct Download from Amazon S3

This is something I might change in the future, but this section describes the initial implementation. When you upload a file, the serverless function handling the request will convert the content to HTML, and then store the generated HTML file on Amazon S3.

One of the goals I had was that content should not be served directly from the serverless functions to avoid incurring high cost, since the pricing for AWS Lambda, and hence Netlify Functions, is based on the number of requests and the duration of the function execution (rounded to the nearest millisecond precision). Furthermore, Amazon S3 is much more suited at serving static files versus AWS Lambda, which is good for our lightweight compute needs.

Another goal of the final implementation was that I wouldn’t allow public access to the S3 bucket.

Let’s go through the possible solutions and see what works and what doesn’t:

  1. Use Amazon CloudFront (Amazon’s CDN) in front of S3
  2. Use Netlify Edge (CDN) in front of S3
  3. Fetch from S3 directly

Using Amazon CloudFront satisfies both goals, since it’s a global CDN and by using something called Origin Access Identity we avoid opening access to our S3 bucket. However, using Amazon CloudFront would add another service to the mix, and since Netlify is a CDN already I didn’t want to introduce another. Next one.

I haven’t been able to find something akin to the Origin Access Identity for Netlify Edge, and therefore solution 2 is ruled out since it would require public access to S3. If there was a way, this would be my ideal solution.

So we are left we solution 3, exposing files from S3 directly. But, didn’t I say that I didn’t want to give public access? 😒 The final approach is that when the serverless function handling the file upload responds, it basically returns a dummy HTML that contains an <iframe> with its src attribute set to a presigned URL for the actual HTML file in S3. This seems like a round-about solution, but it actually satisfies both of my goals nicely. I have total control over who has access to each file and for how long, and I do not expose the file structure on S3 directly.

There is one drawback in this solution, namely S3’s bandwidth cost. Usually for high traffic downloads it would be much cheaper to use a CDN in front of S3 to cache the content, rather than always hitting S3 directly. However, considering that the traffic is going to be close to zero, who cares 🙃 😜

Content Expiration

One of the features I wanted from the beginning was content expiration. Ideal for sharing content that is not ready for prime view or work in-progress, or even stupid notes to a friend.

This feature is implemented using Amazon S3 Expiring Objects. This feature allows you to create a Lifecycle rule where you specify a filename prefix, or specific tags associated with your object, that will trigger its expiration after a given amount of days.

For example, if I stored the content in different virtual directories we could have different prefixes based on the expiration, e.g. s3://my-bucket-name/expiration-days-1/sample-file1.html and s3://my-bucket-name/expiration-weeks-1/sample-file2.html. Then, you can create a Lifecycle Rule specifying that anything under the /expiration-days-1/* should expire after 1 day, and similarly for /expiration-weeks-1/* after 1 week.

In my case, I prefer to use tagging instead of a certain directory layout to stay flexible in the way I name and store files. Therefore, I assign the tag expiration=Days1, or expiration=Weeks1, etc. to each object and create the corresponding Lifecycle rules to check the expiration tag value.

Note that Amazon S3’s expiration is not precise to the minute, or even to the hour. The cleanup runs once a day and therefore an object could be accessible for several hours passed its expiration. For Minibri Temp, this is fine, and we don’t need anything more precise.

URL Generation

The URL generation was probably the most fun feature to develop. The URL could not be very short like URL shortener services, because there is no database to hold the mapping, and just querying S3 until a non-existing filename is found (in case of collisions) would be very slow, and very expensive.

Therefore, the URL should be generated in a way to avoid collisions but still be user-friendly, that is less than 80 characters, and without querying S3. As mentioned before, the generated HTML files are immutable, and once the URL is generated it should always point to the same content.

Right from the start I used the content hash as the URL basis using SHA512, and encoding the hash digest using Base 62, which resulted in roughly 86 characters. This was a bit longer than I liked. Going one step lower and using SHA256 would result in roughly 43 characters which was very nice, but I thought that over time it could lead to several collisions.

In the end, I went with a hybrid approach of using the SHA256 digest encoded in Base 62 as the URL suffix, along with the hour the file was uploaded, and the expiration selected as the prefix. This creates a nice 2-level scoping for each content hash which makes it extremely unlikely to ever hit a collision.

One other reason I have chosen to use the creation date (precision to the hour) is that it automatically sorts the content in the S3 Console, which is a nice bonus while debugging or troubleshooting issues.

Original URL format:
https://temp.minibri.com/view/I65a3dvY1pDbeSQf1NHmZvUR4S3FNszuM5WZyovifwpil9YbMl5NHi3bphnt9H9AfjVxHlRbpUJKmXGXyldVIz

Final URL format:
https://temp.minibri.com/view/c1614636000-e1-rbPYezZ4mwnb3fANNZjBGcwEyNHAsvKDQRR0olVCOIG

Why Netlify?

I play and experiment with many serverless platforms over time, and Netlify is one of my most favourite services at the moment. I wrote an article last year comparing it to Vercel and AWS, but I haven’t used it for anything more complex than just static sites before. I decided to give it a go for a full-fledged application, and I really loved its quality and development experience!

Notable highlights

  • Netlify Dev is essentially a local server simulating what Netlify does in the cloud. It is amazing! 🤩 I set it up to work with Next.js (the static site part) along with my custom Webpack configuration for the serverless functions, and it has really been working fantastically.
  • Pull-request Deploy Previews have been complementary to Netlify Dev in order to fully test the application without affecting the live version in production. For every Github pull request it gives you a URL to access a fully-working application, including the serverless functions which hold the main business logic in my case. Full testing before production achieved! 🏆
  • Rewrites and Redirects is such a subtle feature that you think you don’t need it, until you use it. I have been using AWS Lambda@Edge with Amazon CloudFront to do simple routing for years, but with Netlify redirects and rewrites this becomes a pleasure. For example, in this application when someone visits /create-content, Netlify transparently forwards the request to /.netlify/functions/create-content which is a serverless function handling the request. And you can even use this to proxy requests to servers on different domains.
  • Netlify Functions is another greatly executed feature by the Netlify team. I have used AWS Lambda and Amazon API Gateway for years, inside and outside of Amazon, and avoiding all the hassle of messing with API Gateway to setup basic endpoints is so refreshing. I still build the lambda bundles myself though, rather than letting it go full autopilot in order to control exactly what goes into the bundle.

AWS Resources

As you probably realised at this point, the application needs an Amazon S3 bucket, outside of Netlify’s control. And since this bucket will be into my account, we also need credentials to access it from Netlify Functions, thus we need an IAM User. I provision the AWS resources used by the application using AWS CDK which is great compared to raw CloudFormation, or even other infrastructure-as-code tools.

Total Cost $$$

Depends on how many people will actually use this, but unless it gets hundreds of thousands of uploads per day, it’s not going to cost me more than a Hot Chocolate ☕ per month.

Future Plans

These are just some ideas I might implement in the near, and far, future.

  • Password protected content 🔐
  • Support more source formats, which was one of the original project goals 📋
  • Add an editor to write content directly on the website ✏️
  • Never-expiring content (paid feature?) 💰
  • Multiple files, including images, as part of the same shareable link (paid feature?) 💰
  • Delete after read (remember Inspector Gadget’s self-destructed mission papers?) 💣

Conclusion

My favourite AWS services, AWS Lambda and Amazon S3, along with Netlify’s superb simplicity and great development experience, are all we need to build powerful applications! 🥳 🚀

Changelog

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Goodbye 2020, Hello 2021]]> https://www.lambrospetrou.com/articles/goodbye-2020-hello-2021/ goodbye-2020-hello-2021 Wed, 30 Dec 2020 00:00:00 GMT I don’t usually write end-of-year articles, but 2020 was an eventful year.

2020 Look back

  • Early January 2020 I began preparing for job interviews. Having spent 5 years at Amazon/AWS, it was time to embrace new adventures. After two months of preparation, and one month of sporadic interviews, I accepted an offer from Facebook, and by the end of April I joined WhatsApp. It was definitely a weird, but exciting, time to join a new company during a pandemic lockdown, with all the nuances of onboarding in a new team over video calls 😅 Fun fact, one of my competing offers was from Cloudflare, which I loved as a company and as a team, and I am still debating if I made the right choice… Especially watching their stock price quadrupling in just one year 😱

  • Back in April, I wrote an article about programming languages I wanted to play with during the rest of the year; ClojureScript, ReasonML, and Go. Fast forward to present, I pretty much gave up on ReasonML because it seemed to be a confused language with several rebrandings throughout the year. Maybe I will revisit once the dust settles. ClojureScript is still nice, being a Lisp and all, but unless somebody uses Clojure on the backend as well, it’s much easier sticking to JavaScript or TypeScript on the frontend. Go (or golang) still remains one of my most productive languages. Also, joining WhatsApp meant that I spent a lot of time working with Erlang, and BEAM VM, and since Python is the defacto scripting language in Facebook, that’s two more languages for the year.

  • Around summer, the pandemic really kicked in and I spent most of the time going for walks, when the London weather allowed, but mostly watching Netflix and Amazon Video. Goodbye social life, hello big fat ass. I have to say that there is a lot of great content in the sidelines of these straming services, but there is a lot of garbage as well… I sort of regret spending so many months binge watching some things 😫

  • During summer, the idea of starting my own business/startup was really getting prevalent in my mind. I started reading lots of material and books on small companies, and got really intrigued by the concept of Company of One. This would be my ideal situation in the future, so I still have hope. Some of the best books I read during this period:

  • Entering the last quarter of the year, I finally pulled the plug and decided to launch a side-project I have been thinking for a while. Minibri Score is a sports prediction game that is actually fun; the 10-word pitch line. I was sidetracked a bit on this, but I am planning to release the first version over the next few weeks.

  • By the end of the year, I started getting into and learning applied Machine Learning. I am not overly fond of the underlying theory, and I am mostly interested in the applications and use-cases of Machine Learning. Same way that I am very interested learning new programming languages and using them, but I am not excited by programming language theory or compiler internals. I am mostly answering the questions, what are the most popular algorithms and approaches used, how can I use them, and how can I use them at scale. I look forward to getting more into this next year.

2021 Look forward

  • Programming language wise for 2021 I want to focus on less languages and go a bit deeper. These will be Rust for server backends and WebAssembly, and Python for Machine Learning. If I have time I want to start working with Kotlin again since it’s one of the languages I enjoyed working with a lot in the past.

  • As mentioned in the previous section, I just started reading more about Machine Learning. I already watched some lectures, and read tens of articles, but I want to focus on some specifics over the next few months. I shortlisted my studying to the following, which is subject to change as I learn more:

  • Launch the first version of https://score.minibri.com. Simple. Friendly. Fun.

  • I want to finally convince myself to jump in the exciting world of starting my own business. I have a few ideas in mind, and a few others being discussed with some friends, so it’s just a matter of overcoming my fears mostly. Especially now, since due to the COVID-19 situation, there is not much to lose. The majority of days will be spent locked up inside the house anyway, so why not take advantage of that free time.

  • I read some amazing books in 2020, and I want to do it more in 2021. I plan to reduce the amount of time I spent on streaming, and increase reading time. My kindle library is getting bigger already 📚

  • 2020 was good for my career, but horrible for my fitness. Staying at home the whole day, especially during winter or rainy months was a disaster. Hopefully, in 2021 I will manage to get my ass out of my chair and get active again. I am starting to have built-in swim rings, and it doesn’t look good 🐷

Conclusion

2020 was quite the year for me, mostly in a good way. I am very thankful, and I acknowledge that this was a luxury not many people had this year.

Hoping for an even better 2021, and wishing that health and hope will become the trend of the new year, contrary to the current one.

Be safe, Be healthy 😉

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Life In The UK test — Tips for 2020]]> https://www.lambrospetrou.com/articles/life-in-the-uk-test/ life-in-the-uk-test Tue, 24 Nov 2020 00:00:00 GMT Short version

Long version

The book

I strongly recommend buying the Life in the United Kingdom: A Guide for New Residents, 3rd edition book 📖 Especially if you are like me and prefer reading a “story” rather than just memorizing 100 names and dates taken out of context.

I did a quick read of the whole book cover to cover, and then did a more focused re-read of topics like the law system, or some important leaders, i.e. Queen Victoria, Elizabeth, the wars with the French, etc. (1 day)

The idea is simple, every question on the test is literally a sentence taken out of the book, formed as multiple-choice. Therefore, reading the book at least once gets you 50-60% there.

The only thing I really wish the book provided is a simple timeline with just the names of all the different leaders/queens/kings on the same line in order, otherwise I found it hard to remember all the James and Charles. There is a Henry VIII for God’s sake.

Therefore, I did this stupidly simple timeline myself the night before the exam: https://gist.github.com/lambrospetrou/68241a9243312d95b19812ebf36109df

Feel free to use it, or even submit edit requests if you want to add something missing or fix some inaccuracy. Please note that this not supposed to be a summary of the book, or even a list of all the events. Its main purpose is to list important people, along with “some” events.

Practice tests

Practice makes perfect! - someone wise

After reading the book, your next step is to do practice tests, lots of them. There are many websites, and applications for both iOS and Android, but I suggest using one where you can see why an answer is wrong or correct. This is super important! Also, it’s important to find an app with many tests with a broad coverage of the book chapters.

I used the Life in the UK Test 2020 Android application, and it was immensely helpful to see the exact exerpt from the book that justifies why the answer is correct or wrong. Reading those sentences over and over, even the ones you already know, will just stick them in your brain for good 🏋🏽‍♀️ This application has a “cramming mode”, where it shows you the answer before-hand. I did not use that though, because putting some effort every time to make a choice forces you to think, and this forces your brain to start doing associations between names, dates, and events. Reading the answer will just magnify that reasoning.

The above application has 48 tests as of the time of writing. This was my practice methodology:

  1. Did every test in order, but before moving to the next one, I repeated the test until I scored at least 22 out of 24. Anything less, and I was doing the test again.
  2. Once I did all the tests, I started doing them again in random order. However, same as before, if I scored below 22/24 I repeated the test.

Note that there are no difficulty levels in this test. All of the questions are pretty much of the same difficulty depending on what you are good at remembering. For example, I am horrible at remembering all the sportswomen/men and poets, so I came up with memorable tricks to remember at least some of them, either to help pick the correct answer, or to eliminate the wrong ones. This memorable information does not have to be anything scientific. I am even proud to say that for some of them I associated the sound of their name to their invention or sport 😅

Overall, I spent roughly 6 hours just doing tests. The night before the exam I was at a point of completing a full test within 2 minutes, making just 1-2 mistakes, if any at all.

Conclusion

Read the book, do a hell lot of tests, and use your imagination to remember stuff.

Good Luck 🥳 🙌

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Minibri Score — The Inception]]> https://www.lambrospetrou.com/articles/minibri-score-the-inception/ minibri-score-the-inception Sun, 25 Oct 2020 00:00:00 GMT TL;DR

Minibri Score is a sports prediction game that is actually fun!

Visit https://score.minibri.com to signup for early access 🚀 and stay tuned for its release ⏳

Why?

Whenever there was a big football event, like UEFA Euro 2016 and FIFA World Cup, me and my friends were doing contests on who can predict the score, top scorers, etc.

We were using a Google Spreadsheet (or Excel) with some formulas, and in some cases extended it with a bit of JavaScript code to calculate our special point awards and generate a ranking table. At the end of the tournament the player with the most points was the winner (you can put money down to make it more interesting 💸 but don’t get too addicted…).

The funny thing is that this is something that many groups of friends do, and it’s even popular among colleagues in companies, since it’s a nice bonding activity. Who doesn’t like some healthy competition!

Three years ago, one of my friends had the idea of taking what we were doing on spreadsheets, and make a product out of it. It didn’t seem exciting to me at first; who was going to use it, and who was even going to pay for it. We did some research and there were not a lot of similar alternatives on the market. So, we spend the next few weeks discussing the product on and off. After a few weeks of inactivity it basically went to sleep.

A few months ago, I wanted to start something on my own, outside of work, a side-project that I was going to have fun working with. If it got any traction, and customers used it, even better. Therefore, I revived that original idea, refined it, and decided to implement it. The timing now is actually quite good, since next year there will be several big sports events, and it will be a great opportunity to use it, even if just among my friends.

What?

The idea behind Minibri Score is not new, it’s not innovative, and it does not have to be.

I spent a few weeks looking for alternatives, competitors, but apart from 1 or 2 others, everything else is way too bloated. Full of advertisements all over the screen, a million menus and complex options, and most of them really focused on just a couple of sports. They all try to be another fantasy league. For context, a fantasy league is where you basically focus into a specific league tournament (e.g. Premier League in England), and choose/buy/sell players, take points every time the players you chose score, or their teams win, etc.

Minibri Score is not a fantasy league. I don’t want it to be one.

The only thing a user will have to do is predict the outcome of a fixture. All of the fun, and strategy, revolves around the point awards. There are several ways to get points, and they all use the score prediction versus the final score in some sort. Maybe you have to take a big risk with a low-chance score to win the big points, or always play safe and getting steady points.

Which Sport?

One of the major drawbacks of the alternatives, which is a consequence of going with a fantasy league style of tournament, is the fact that you can only find a couple sports supported, mainly football. There is just too much work into the specifics of a single sport.

One of the main goals I have with Minibri Score is to allow any two-sided sport to be supported right from the start. There are several things that are sport specific, for example volleyball has sets instead of just a final score. There will be exciting point awards around the sport specifics, but the core functionality is going to be shared across all sports.

👉🏽 As long as it’s a two-sided sport, Minibri Score just works 🙌🏽

I consider this a big differentiator, and an advantage since many people will be able to have fun. You will be able to compete in a tournament with teams from your local town that you would never find in major fantasy leagues. Or even create a tournament for a sport that is not as popular, but still follows the same principles of two sides and a score.

Simple. Friendly. Fun.

I want a product that is:

  • Simple to use without complex menus and unnecessary options.
  • Friendly, without advertisements and flashy popups coming out from everywhere.
  • Fun among friends and colleagues alike.

When?

I am hoping to release the first version before the end of the year 🥳

Next year is going to be exciting 🎾 🏓 🏏 ⛹️‍♀️ ⚽️

👉🏿 Visit https://score.minibri.com to signup and I will email you when it’s out and ready 🚀

You can also follow its development on Twitter at https://twitter.com/MinibriScore 🐦

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Simple and Low-cost Investing guide (for the UK)]]> https://www.lambrospetrou.com/articles/simple-low-cost-investing-in-uk/ simple-low-cost-investing-in-uk Sun, 04 Oct 2020 00:00:00 GMT Over the past few years I read a lot of articles, blog posts, and a few books on stock market investing. I wanted a simple approach that would have good enough returns, but without spending lots of time maintaining a portfolio and risking losing money just because I was a few hours late selling some stock.

In this article I will describe what I do as of today, and why I think it’s good enough for anyone that also strives for a simple low-maintenance approach with good returns long-term (10+ years).

TL;DR - The short version

  1. Max your company’s pension contributions to get their maximum rate, it’s free money.
  2. Open a Stocks and Shares ISA account.
  3. Setup a monthly direct debit to invest in low-cost, low-maintenance index funds like the FTSE Global All Cap Index Fund and FTSE Developed World ex-U.K. Equity Index Fund inside your ISA account.
  4. Save money in NS&I Premium Bonds, and win monthly tax-free prizes.
  5. If you still have money after maxing out all the previous options, then awesome for you, keep investing in global index funds using a regular investing/trading account.

That’s it, without too much effort we maximized our long-term returns in a relatively low-risk, low-cost approach 💸

Stocks and Bonds

The most important book I read which I recommend everyone to also read is The Little Book of Common Sense Investing. The main idea is that to avoid most of the risk, albeit reducing potential returns, is to invest in the whole stock market using low-cost index funds. This approach exposes you to the entirety of traded companies, which means that if a company goes under water it won’t affect you a lot, since some other company that did well will cancel out the losses.

Based on this simple idea, my current portfolio includes:

  • U.S. Equity Index Fund - This fund invests in the top (roughly) 3500 companies in the US. This fund is very simple, thus having very low annual maintenance cost at 0.10%. This is the UK version of the super famous US fund Vanguard Total Stock Market Index Fund - VTSAX.
  • FTSE Global All Cap Index Fund - This fund invests in the top (roughly) 7000 companies in developed and emerging markets around the world. The annual maintenance cost is a bit higher on this fund (0.23%) but still well below the market average for actively-maintained funds. This is the UK version of the US fund Vanguard Total World Stock Index Fund - VTWAX. Note that this fund is somewhat a superset of the above, and just using this one can be enough and simpler, depending on if you want exposure outside the US.
    • FTSE Developed World ex-U.K. Equity Index Fund - Update 2023: I switched from the Global All Cap Index fund to the Developed World Index fund. It excludes the Emerging Markets which had dissapointing growth over the recent years, and I prefer the more stable developed world. This fund also excludes the UK to avoid home-bias, which is OK for me, since living in the UK and getting paid in British Pounds (£) is enough exposure to the UK. It has a lower annual maintenance cost too at 0.14%.

Both funds are relatively new (just a few years old) which explains why their total invested assets are way below the corresponding ones in the US (hundreds of billions), but hopefully as they get bigger, their cost will get further down, and maybe even reach the US-levels at some point 😅

All reputable investing resources suggest that someone invests in bonds as well, which have much lower returns but also much lower risk. I personally don’t invest in bonds, since I have several decades of investing ahead of me, so I don’t need the lower risk at the moment. But, if you want to have some exposure in bonds, the following is a good global fund.

  • Global Bond Index Fund - This fund invests in more than 12000 bonds from around the world, with an annual maintenance cost of 0.15%.

There are more things to consider, but by investing in the above funds you are exposed to the whole market at a low cost, and with good enough potential returns.

Honestly, you could simplify this even further. If you really want a no-maintenance approach, then you can just invest in one of the LifeStrategy Funds (or similar alternatives) and let the fund managers figure out which funds to invest in. You can choose the pecentage you want to be invested in stocks, the rest is invested in bonds, depending on your risk appetite. For example, the LifeStrategy 40% Fund invests 40% of your money in stocks, and 60% in bonds. Therefore, the easiest approach would be to invest in the LifeStrategy 80% or 100% funds early in your career, and switch to the 20% or 40% funds closer to your retirement.

I use the Vanguard funds because I find them to be among the cheapest and the best around, but keep in mind that most of the investment companies and brokers provide their own variations and alternatives, so do your own research when choosing a broker.

US versus International

As I already mentioned, the U.S. Equity Index Fund (like VTSAX) is only investing in US companies, whereas the FTSE Global All Cap Index Fund (like VTWAX) invests in international markets as well.

There are huge and myriad debates among professionals and individual investors trying to decide what’s the best. Trying to answer the question of how much you should invest internationally, and if you even need to. Each side comes up with all sorts of (valid) arguments supporting their view. You can read some of the discussions at [1] [2] [3] [4] [5] [6].

Personally, I believe that it makes sense to have international exposure. However, I find the US market stronger at the moment, hence why I invest in both funds, so that I can tilt my weight more on the US market. Having said that, I gradually increase my share of the global fund because I believe that non-US markets will thrive over the next few years, maybe even more than the US.

This is just another speculation, my speculation, thus if you have no opinion just go with the global fund and relax.

Stocks and Shares ISA

In the UK there are some special savings accounts called Individual Savings Accounts (ISA) which allow you to save up to £20000 a year (as of 2020), and any interest you get is tax-free.

The great part is that you can have a Stocks and Shares ISA account, which means that you can invest in the funds mentioned above and any return you get is going to be tax-free. As you can imagine, over the years, the reinvestment of compound tax-free interest can result in substantial money 💰

In my opinion, everyone should use a Stocks and Shares ISA, and if they exceed the yearly limit, then proceed with a normal stock trading account.

Lifetime ISA - LISA

I haven’t personally used this (yet), but if you are planning on buying a home in the UK, or want to plan for retirement it could be a nice addition to your normal ISA account. You can put up to £4000 into a LISA account, and the government will add an additional 25% of your investment up to £1000, only applicable till you are 50 years old.

The £4000 comes out of the total ISA allowance, which is why it might not be useful for someone not planning on staying in the UK for long, since the only way to get the money out of the LISA account is to either buy a house, or withdraw the money once you reach 60 years old.

NS&I Premium Bonds

I recently found out about the NS&I Premium Bonds, which is a 100% safe way to store your money since they are backed by the government, and they have monthly money draws that give out tax-free prizes.

There is no guaranteed return on the premium bonds, but for every £1 you invest you get one entry in the monthly draw. Prizes range from £25 to millions, and therefore if you are lucky it can be very beneficial.

Keep in mind that unless you win the big prizes this is not going to match the returns of the stock market. However, in the few months I have been using this, I got more returns than any savings account rate a traditional bank would give, which makes this a winning option for me.

Bank Savings accounts

A few years ago you could find traditional bank savings accounts, or even current accounts, with high interest rates in returns. For example, the Lloyds Club Current Account had 5% return up to a £5000 balance, which over the years got reduced down to 4%, 3%, and now it’s around 1%.

However, I can’t find any savings account these days that’s worth saving money in, therefore I recommend just putting a small amount of money into your Stocks and Shares ISA using one of the above low-maintenance funds instead.

Stock Trading Brokers

I am not writing this article to advertise a specific broker, even though I solely provided links to Vanguard 😅

The only suggestion I have is to go with a broker that has great customer support, and low annual maintenance costs. These are the two things I care about, especially since I want a low-cost approach, and a good customer support is always helpful when I have questions.

As an example, Vanguard’s low fees, clear costs approach is one of the things I love about it. There is an annual account cost of 0.15%, and in addition to that the annual maintenance cost for the funds invested. So far, 0.15% account management cost is the lowest I have found, hence why I use and recommend them.

One low account fee Just 0.15% per year

Capped at £375 per year for accounts over £250,000

— By Vanguard Investor

As of 2023, Interactive Brokers also introduced their ISA accounts, which has even lower fees. It chargers £3 / €3 per trade for Western European stocks, with a minimum £3 charge per month.

£3 / €3 per trade for Western European stocks, with no added spreads, account minimums or platform fees. These simplified commission rates are available with IB SmartRoutingSM, which optimizes the execution quality for clients by accessing the many exchanges and trading venues across the continent.1 Pricing on US stocks starts at just USD 0.005 per share.

There is a minimum monthly activity fee of £3 for a Stocks and Shares (adult) ISA and £1 for a JISA. You receive one free withdrawal per month and there are no custody fees for all account types. All ISA accounts are cash only, no margin.

— By Interactive Brokers

Conclusion

No matter how much money you can save per month, even if it’s just £50-100, it’s highly worth it to open up a Stocks and Shares ISA account, and invest that money into one of the low-cost, low-maintenance, global funds mentioned above (or similar alternatives) and enjoy the power of tax-free compound interest. 💸⛱️💰🏡

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[V8 Isolates for fast JavaScript execution in Go]]> https://www.lambrospetrou.com/articles/golang-v8-isolates/ golang-v8-isolates Sat, 26 Sep 2020 00:00:00 GMT I am working on a side project similar to Leetcode/HackerRank (more on that in another article) where I have to execute some given JavaScript code against hundreds of inputs. The first approach I thought was spawning a subprocess and executing node -e "<code-to-evaluate>" directly passing the script I want to run. This works fine and gets the job done in a low throughput scenario.

However, I remembered that Cloudflare Workers, Cloudflare’s serverless offering, is using V8 Isolates to execute the submitted JavaScript code [1][2][3]. Therefore, I wanted to see how much faster that can be for my usecase.

Experiment

I have written a very basic experiment where the JavaScript code calculates the sum of a given integer array (and some extra arguments). The code is as simple as below:

// f = (a: number[], b: number, c: string) => number
const f = (a, b, c) => {
  return a.reduce((c, acc) => acc + c, b) + c.length;
};
result = f(%+v, %+v, %+v);
result;

The weird looking arguments, %+v, are just placeholders which will be replaced in my Go program with actual values. Remember, the reason I wanted this in the first place is to run some code (in this case the function f) against lots of input arguments.

The last line evaluates to the value of result, just imagine that this is interpreted line by line.

Spawn Node

The simplest approach I thought was to invoke node passing the JavaScript code to be evaluated.

func node(script string) string {
  output, e := exec.Command("node", "-e", script+"\nconsole.log(result)").Output()
  if e != nil {
    log.Fatalf("Error: %+v\n", e)
  }
  return string(output)
}

The script+"\nconsole.log(result)" might seem strange at first. We need to suffix our script with console.log(result) since we capture the output of the subprocess.

V8 Isolates

I won’t go into the details of what a V8 Isolate is other than quoting Cloudflare’s documentation…

V8 orchestrates isolates: lightweight contexts that group variables with the code allowed to mutate them. You could even consider an isolate a “sandbox” for your function to run in.

A single runtime can run hundreds or thousands of isolates, seamlessly switching between them. Each isolate’s memory is completely isolated, so each piece of code is protected from other untrusted or user-written code on the runtime. Isolates are also designed to start very quickly. Instead of creating a virtual machine for each function, an isolate is created within an existing environment. This model eliminates the cold starts of the virtual machine model.

And Chromium’s documentation.

An isolate is a concept of an instance in V8. In Blink, isolates and threads are in 1:1 relationship. One isolate is associated with the main thread. One isolate is associated with one worker thread.

A context is a concept of a global variable scope in V8. Roughly speaking, one window object corresponds to one context.

Let’s see how we create and use a V8 Isolate in Go, using https://github.com/rogchap/v8go.

func v8isolates(script string, isolateOpt ...*v8go.Isolate) string {
  var isolate *v8go.Isolate
  if len(isolateOpt) > 0 {
    isolate = isolateOpt[0]
  }
  ctx, _ := v8go.NewContext(isolate) // Passing `nil` creates a new Isolate
  defer ctx.Close()
  output, e := ctx.RunScript(script, "function.js")
  if e != nil {
    log.Fatalf("Error: %+v\n", e)
  }
  return output.String()
}

The main method we use is ctx.RunScript(script, filename) which accepts the code to execute (argument script), and a (fake) filename. The filename will be used inside the generated error stacktrace in case the execution of the passed script fails for any reason. The filename itself does not need to actually exist on the filesystem. The returned value of ctx.RunScript(...) is the last evaluated expression of the given script, which explains the last line in our JavaScript code above which is just result.

Results

Cloudflare makes bold statements about the performance of their Workers due to using Isolates, but I was still mind-blown by how fast they actually work 🤯

I wrote some basic benchmarks for the above two functions and used a simple input for the array (10 to 150 integer numbers).

Macbook 16” 2020 (16-threads)

➜ go test -bench . -benchtime 1s -benchmem
goos: darwin
goarch: amd64
pkg: github.com/lambrospetrou/code-playground/golang-v8isolates
BenchmarkNode-16                              19      62746119 ns/op     43208 B/op     57 allocs/op
BenchmarkNodeParallel-16                     176       6861996 ns/op     43142 B/op     55 allocs/op
BenchmarkV8IsolatesReuse-16                 4171        290742 ns/op        75 B/op      6 allocs/op
BenchmarkV8IsolatesNoReuse-16                823       1414748 ns/op        88 B/op      7 allocs/op
BenchmarkV8IsolatesReuseParallel-16        30026         45703 ns/op        75 B/op      6 allocs/op
BenchmarkV8IsolatesNoReuseParallel-16       2079        983084 ns/op        83 B/op      7 allocs/op
PASS
ok      github.com/lambrospetrou/code-playground/golang-v8isolates    10.218s

Surface Pro 2017 (4-threads)

$ go test -bench . -benchtime 1s -benchmem
goos: linux
goarch: amd64
pkg: github.com/lambrospetrou/code-playground/golang-v8isolates
BenchmarkNode-4                            27    47009289 ns/op      51359 B/op     92 allocs/op
BenchmarkNodeParallel-4                    76    15338893 ns/op      51548 B/op     92 allocs/op
BenchmarkV8IsolatesReuse-4               3342      385395 ns/op         74 B/op      6 allocs/op
BenchmarkV8IsolatesNoReuse-4              692     1956912 ns/op         84 B/op      7 allocs/op
BenchmarkV8IsolatesReuseParallel-4       5684      213542 ns/op         75 B/op      6 allocs/op
BenchmarkV8IsolatesNoReuseParallel-4      303     3631768 ns/op         83 B/op      7 allocs/op
PASS
ok    github.com/lambrospetrou/code-playground/golang-v8isolates      12.725s

The difference between BenchmarkV8IsolatesReuse and BenchmarkV8IsolatesNoReuse is that instead of creating a new Isolate per test run, we use one Isolate per thread and only create a new context per run.

As you can see, there is a huge performance boost when using V8 Isolates. At least one order of magnitude faster when creating new Isolate per run, and two orders of magnitude with Isolate reuse. It’s not unexpected since spawning a process is quite expensive but still…

We can do a lot of optimizations to how we use the spawned process as well. For example, we can spawn one node process per thread and then communicate with it over standard input/output, and having a small script that reads the standard input, evaluates it, and prints the result on standard output. This will probably be even faster than the V8 Isolates but apart from the fact that we completely lose execution isolation, it’s also a lot more work, so it’s out of scope.

You can find the code for the experiment at https://github.com/lambrospetrou/code-playground/tree/master/golang-v8isolates.

Conclusion

V8 Isolates are amazing 🚀

However, if you are going to use them on a real production multi-tenant system make sure to secure it further (read references below).

References

  1. https://developers.cloudflare.com/workers/learning/how-workers-works#isolates
  2. https://blog.cloudflare.com/cloud-computing-without-containers/
  3. https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[AWS Lambda and SQLite3 over Amazon EFS]]> https://www.lambrospetrou.com/articles/aws-lambda-and-sqlite-over-efs/ aws-lambda-and-sqlite-over-efs Sat, 11 Jul 2020 00:00:00 GMT I am a huge fan of AWS Lambda and serverless in general. I also love the reliability, speed, and simplicity of SQLite, and I was looking to find a few projects to use it.

The issue was that until recently you could not use AWS Lambda together with SQLite, other than bundling the whole database file into the Lambda bundle zip file itself. In practice this meant you only had a readonly database though, since each concurrent Lambda invocation would have its own copy of the database.

However, last month, AWS released integration between AWS Lambda and Amazon EFS. Since Amazon EFS already supported NFSv4 lock upgrading/downgrading which is needed by SQLite, it means that now we can access an SQLite database file stored on an EFS filesystem through AWS Lambda in a read-write mode 🔥 🚀

Quick Test

I did a quick test using a Node.js Lambda function and it seems that the EFS connectivity adds around 100ms of latency overhead to the Lambda invocations when accessing the database over EFS versus a local filesystem. This test is by no means scientific and performance will vary a lot depending on your database file size, the concurrent connections you have, and the number of concurrent writes to the database.

The simple test application (code available) executes 100 insert statements using a transaction, a select query with a where condition, and then a select * query.

AWS Lambda

sqlite-aws-lambda-efs

The execution time of the statements accessing the SQLite database is roughly 140ms. There is some variation but that’s what I was averaging after 10+ runs. I also tried several AWS Lambda memory configurations to see if the more powerful CPU would affect the latency in any way but it didn’t. This makes me think that the overhead is pure network IO.

Local - Surface Pro

sqlite-local

The execution time of the statements accessing the SQLite database is roughly 30-40ms. Some variation expected here as well albeit lower since it’s a local filesystem.

Is this Production ready?

Most probably not…

The correct answer is it depends!

  • Accessing an EFS filesystem adds latency to your Lambda invocations just due to the network overhead.
  • SQLite was designed to be the best in-process database in the world. SQLite’s website itself describes amazingly well the appropriate uses for SQLite, including the inappropriate ones. Our experiment, since it introduces a network between the application (Lambda code) and the database file (EFS) falls into the inappropriate uses…
  • SQLite supports concurrent readers, but only one writer can be writing to the database, therefore only one Lambda invocation will be able to write to the database at any point in time which is going to be a throughput bottleneck.

The above facts make it obvious that using SQLite with AWS Lambda is a very bad idea for production systems with thousands of users, or high-throughput of writes, or any system that needs low latency for that matter.

However, for side projects, small team internal projects, applications where latency is not an issue, or situations when you want to build a quick demo prototype it’s probably alright.

I am sure the AWS Lambda users will find interesting ways to abuse this integration, but that’s expected, and it’s half the fun 😜

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Make it Work, Make it Beautiful, Make it Fast]]> https://www.lambrospetrou.com/articles/make-it-work-make-it-beautiful-make-it-fast/ make-it-work-make-it-beautiful-make-it-fast Sat, 04 Jul 2020 00:00:00 GMT

The OTP team is respecting the old slogan: first make it work, then make it beautiful, and only if you need to, make it fast.

I was reading the Learn You Some Erlang book this morning and at some point I stumbled upon the above sentence. I heard this phrase online for the first time about 2 years ago from Jose Valim (creator of Elixir) during one of his Advent of Code videos.

This phrase really resonates with me because this is exactly how I approach writing code and solving problems as well. Let’s elaborate a bit more…

Make it work

The main reason we write code (other than fun) is to solve a problem, and if our solution is not correct, then we didn’t do our job. Correctness comes first!

During this step, I usually write code without too much thinking into abstractions, how to design the components, function names, or other appealing details. I just want to solve the problem.

Make it beautiful

This step can be intepreted in many ways depending on the programming language used, the domain of the problem, your team, etc.

However, it could be refactoring the public API of your library/module, coming up with the right abstractions, splitting long functions into smaller more testable ones, and anything else that makes the code readable, extensible, and ready to be reviewed by your peers.

Make it fast

This step can be considered optional in a lot of day-to-day cases. However, it’s always good practice, and over time it can become a habit to check your code for performance issues.

You solved the problem correctly (step 1), and the code is all pretty and ready to be reviewed (step 2), but is it fast enough? An even better question would be: Is it slow enough to cause problems?

The time and effort you put into making your solution fast solely depends on your use-case, but always remember to ask this question.

Conclusion

Always follow this approach when writing code:

  1. Make it work
  2. Make it beautiful
  3. Make it fast
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Enjoyable browser automation with Puppeteer and Playwright]]> https://www.lambrospetrou.com/articles/enjoyable-browser-automation-puppeteer-playwright/ enjoyable-browser-automation-puppeteer-playwright Sat, 16 May 2020 00:00:00 GMT I have been doing a lot of web development over the past few years, both in my own time with my side projects but also at work. One of the things that you need at some point is to write a script to automate some kind of action usually done in the browser. For example, this could be navigating to certain pages and taking screenshots, or automated form submission, or automated UI testing.

One of the simplest use-cases I have for my own website is to generate a PDF version of my CV page. There are several reasons why a PDF file could be better sometimes than a website, so it’s useful to have one in hand. Since I already had the HTML version, what I used to do for several years was to use Chrome’s Print to PDF functionality.

However, a couple of years ago, Google released Puppeteer which allows someone to script a headless version of Chromium. From the second I saw it I knew it was going to be big! Easy, simple and headless browser automation through Node. The potential use-cases are infinite 🛸

Since then it has been used by companies to make amazing things.

  • Automated robust UI tests inside headless Chrome, instead of the finicky Webdriver approaches that were the standard back then.
  • Performance profiling and benchmarking of web applications.
  • Scripting of manual tasks that people were doing like form submission.
  • Web crawling through a normal browser to avoid issues with previous approaches that were using browser-like fakes.
  • Server-side rendering of dynamic single page JavaScript application to avoid SEO issues.
  • Many other things that required an actual browser running a website.

One of the problems with Puppeteer is that it only supports Chromium so far.

I was extremely happy to see a few days ago that Microsoft finally released the v1 version of Playwright. Playwright is extremely similar to Puppeteer, but it extends its amazing feature-set by supporting all the three major browsers, Chromium, Firefox, and Webkit 😍 Not only that, but they added very cool features like auto-waiting for element selection which greatly simplifies DOM manipulation.

There are a few features that are not supported by all the browsers, so keep that in mind if something fails in your scripts, make sure that the browser you use supports it.

Code

From the little time I spent with Playwright its API seems extremely similar to Puppeteer’s so it’s easy for someone to jump between the two or migrate from one to the other.

As an example, see below an excerpt of the code I use to generate the PDF version of my website.

The full code can be found at my CV repository.

Local server

While I am making changes to my CV I want to be able to generate the PDF and test it without having to push my changes to the live website. The following code starts a local server at port 12345 that serves the assets from inside the build/ directory.

const handler = require('serve-handler');
const http = require('http');
const path = require('path');
const WS_BUILD=path.join(__dirname, './build');

const server = http.createServer((request, response) => {
    return handler(request, response, {
        public: WS_BUILD
    });
})
server.listen(12345)

Puppeteer

The following code is what we need to make Puppeteer visit our local server, wait for the page to fully load, and then generate a PDF named cv.pdf.

const puppeteer = require('puppeteer');

async function generatePdfPuppeteer() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('http://127.0.0.1:12345', {waitUntil: 'networkidle2'});
    await page.pdf({
        path: path.join(WS_BUILD, 'cv.pdf'), 
        format: 'A4',
        margin: {
            top: '0.39in',
            left: '0.39in',
            right: '0.38in',
            bottom: '0.38in'
        }
    });
    await browser.close();
}

Playwright

The code for using Playwright is (almost) identical to Puppeteer in our simple use-case.

const { chromium } = require('playwright');

async function generatePdfPlaywright() {
    const browser = await chromium.launch();
    const ctx = await browser.newContext();
    const page = await ctx.newPage();
    await page.goto('http://127.0.0.1:12345', {waitUntil: 'networkidle'});
    await page.pdf({
        path: path.join(WS_BUILD, `cv-playwright.pdf`), 
        format: 'A4',
        margin: {
            top: '0.39in',
            left: '0.39in',
            right: '0.38in',
            bottom: '0.38in'
        }
    });
    await browser.close();
}

Issues with WSL on Windows 10

I am using the Windows Subsystem for Linux (WSL) version 1, and I had issues while running both Puppeteer and PlayWright due to some incompatibilities with the browser binaries.

You can find more information for this issue in these discussions:

To make Chromium work the arguments --no-sandbox, --disable-setuid-sandbox, and --single-process have to be passed to the launch() method. See the modified lines below.

Puppeteer

const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox', '--single-process']});

Playwright

const browser = await chromium.launch({args: ['--no-sandbox', '--disable-setuid-sandbox', '--single-process']});

Conclusion

Having the ability to write a few simple lines of code and automate a full-blown browser is amazing. What I discussed above only scratches the surface of what can be done and I am always excited to see how people use these stuff.

Enjoy 😉

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[The worklog format 1.0]]> https://www.lambrospetrou.com/articles/the-worklog-format-1/ the-worklog-format-1 Tue, 12 May 2020 00:00:00 GMT Last week I wrote about note taking being my productivity tool, and then about the worklog being my most important daily document.

Since then, I had several friends and colleagues reaching out asking specifics about the worklog’s format and how I structure its content. I will try to briefly describe what I use hoping that it can help others as well.

Before we dive in, I would like to reiterate once again that I don’t really believe in the one size fits all idea, and what works for me might not work for you. However, seeing the structure I use might inspire you to find one that works for you.

What?

First of all, I urge you to read the original article explaining what the worklog is to better understand the rest of this article.

The worklog contains essentially three things:

  1. Meeting notes
  • Who was in the meeting
  • When was the meeting
  • What was said during the meeting
  1. Activity log
  • What did I do in each day
  1. Todo items
  • Tasks for the next 1-3 weeks (quite fine-grained)
  • Couple of long-term things that I need to break down

How?

I use the following format for my worklog file:

# Worklog 2020-01-01 to present

## Meetings

### 1:1 with manager <userxxx@abc.com> @ 2020-05-10

- topic 1
- topic 2
- action item A
- action item B
- ...

### Meeting for feature XXX @ 2020-04-29

- Who
  + nameA <usera@abc.com>
  + nameB <userb@abc.com>
- thing 1
- thing 2
- action item A
- agreements
  + a
  + b
- open questions...

## Activity log

### 2020-04-29

- Meeting for XXX
  - Notes: <URL>
- Worked on feature Y
  - <Task URL> 
  - <Code review URL> implements ZZZ

### 2020-04-30

- Watched training videos
  + infra: <link>
  + networking
- Investigated YYY
  + <link to the file containing details or to the internal issue>
.
.
.

## TODO

- implement DDD
  + <internal issue link>
- training videos
  + <link>
- do a deep dive on XXX

As you see, it’s quite simple. Three separate ordered lists.

You will notice that the meetings section orders the meetings by date in descending order (latest at the top), whereas the activity log section is ordered by date in ascending order (latest at the bottom).

This is super handy for me because over time the file grows, and in my productive months it can get hundreds of lines—each worklog file spans my tenure in a team/company (can be several months or years long). This structure allows me to instantly scroll to the top and start writing about the current meeting, or scroll to the bottom and start writing about today’s activities, without wasting time figuring out where I need to write. Almost all the text editors and IDEs have shortcuts to move at the top and bottom of a file which makes this approach efficient.

Regarding the todo section, I try to keep it short and not have more than 10 items, since at that point it becomes a backlog. I want items that I will work on during the next 2-3 weeks ordered in priority, with top being the next one to be done. Having said that, I almost always find myself having 1-2 items that are long-term oriented, for which I need to do some kind of investigation. It’s useful to have them here to remind myself to do a bit of work every day.

Every morning, I open the worklog, remind myself what needs to be done by going over the todo section, and I make sure it is synced with my team’s backlog in case priorities shifted.

That’s it folks, I hope this is helpful!

Changelog

  • 2023-04-02: Updated the format of the activity log to use separate sections per day.
]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Best tip I received — The worklog]]> https://www.lambrospetrou.com/articles/best-tip-the-worklog/ best-tip-the-worklog Sat, 09 May 2020 00:00:00 GMT In a previous article where I wrote about my favourite productivity tool I briefly mentioned about my daily work log habbit. This article is its origin story.

Origin

About 5 years ago I returned to Amazon (did an internship the year before) as a full-time employee after finishing my Masters degree. During one of my first 1:1 meetings with my manager we were talking about career growth and to this day I remember him saying the following.

If I can only give you one tip, it’s to keep notes of what you do every day. Not for the usual reasons like performance reviews, but because there will be days that you will feel unproductive, thinking that you don’t produce anything, and it will make you feel bad. Those notes will be the only way to go back and see all the work you did during those days.

This is by far the best tip I received in my (albeit short) career, and it’s the first tip I give to every new hire joining my team as well.

I call these notes the worklog, and it’s essentially a big text file with everything I do throughout the day in chronological order. Anything can go into the worklog like tasks or issues I worked on, code reviews I opened, meetings I attended (including notes), or research I do on new features.

My rule is that anything taking away my time should go into the worklog.

What’s the big deal?

I consider the worklog to be the most important document I keep regarding work.

  • I use this worklog for every performance review and/or promotion document preparation since I can literally scan through it and pull out a list of the important stuff I did for a certain period amount of time.
  • I cannot count the times I grepped the worklog for things I remember I did in the past, but vaguely remember exactly what it was or how it was solved. It’s been invaluable!
  • Occasionally, a colleague will ask about something I did and the answer will be in the worklog since information is spread in several internal tools making discovery hard.
  • Those rainy days I mentioned above will come no matter how genius you are, and the worklog will be there for you.

Want a tip?

Keep notes of everything you do throughout the day. Embrace the worklog!

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[My digital braindump and productivity tools]]> https://www.lambrospetrou.com/articles/digital-braindump-and-productivity-tools/ digital-braindump-and-productivity-tools Fri, 08 May 2020 00:00:00 GMT I thought about writing this article many times over the years but I always thought it was not really something worthy of writing. However, after I saw the announcement for Github Codespaces a few days ago I decided to write it because that announcement resonated so much with my daily workflow.

I will describe what I have been using for the past 5 years as my productivity tool, to keep my thoughts, notes, and ideas organised.

What do I write down?

See below a non-exchaustive list of things that I take notes for over time:

  • I write down everything I do during my day at work (extremely crucial and deserves an article on its own)
  • Draft articles for my website
  • Ideas for side projects
  • Business ideas I think (and then reject)
  • Material to read, watch or listen for several subjects
  • Research I do on investments
  • Notes for flats, houses, jobs when I am in the lookout
  • Many many more things…

I am pretty sure everyone encounters at least a few of the items listed throughout the day.

There are many productivity tools available, especially mobile apps, that are supposed to simplify your life, but almost everything I used, and I tried hundreds of them, always has something that ruins it for me.

As a software engineer I spend a lot of time in text editors working with source code. At some point I realised that the best productivity tool was sitting right in front of me… That is text files.

In the old days people were using pen and paper to take notes, and many still do, including myself! Pen and paper is easy, quick, and everywhere.

In addition, one of the requirements of any productivity workflow is to have your notes available on all of your devices. The best ideas always come at the weirdest time and place.

What do I use?

A text file is very similar with pen and paper. Text file editing is available on every device type and it’s quick and easy to write anything.

File synchronization across devices is a solved problem nowadays with services like Google Drive and Dropbox. In the software world we also have source code versioning software like Git.

Many of the applications available restrict you in the type of notes you can take, the format to write them in, searchability might be limited, they have size limit restrictions, and they might not even support your device or operating system.

Having said all this… What do I use?

My productivity tool is basically a directory named notes consisting of several text files, organized sometimes in other sub-directories. Initially, I had this directory inside Dropbox, then Google Drive, and now in a self-hosted git repository.

I love this approach because not only it’s extremely easy to write down anything at any time, but I can also put additional files or material inside that directory without any restriction. All text editors provide amazing search capabilities, from simple file name searching, to fuzzy text search of the content of your notes.

You can write your notes in formats like Markdown or AsciiDoc and have your editor render them nicely. Many editors also support displaying images or PDFs inline, and you have the full power of your device to open any type of file inside this directory. Again, there is no restriction of what files to put in your directory.

So, long story short:

  • I use Visual Studio Code as my editor rooted at the notes directory.
  • I write most of my notes in Markdown (.md) format.
  • I have one file named scratchpad.md which acts as my temporary canvas. This is probably my most edited file. I do anything temporary in here, from drafting medium-to-long emails before putting them in Outlook, writing long messages before pasting them in Slack/Mattermost/Facebook, and whatever else that needs a temporary space.
  • I have my passwords inside encrypted files.
  • I use a self-hosted git repository, but I am considering moving to a Github private repository. Using git is powerful for me because I can immediately jump to any previous version of any file in the directory from the beginning of time.

What else?

The approach described above covers almost the entirety of my needs. However, there are two things that I do in other tools.

  1. Tiny notes that I need to take while on the move, i.e. walking to the train or the bus, or standing in the elevator, etc. I use Google Keep for this but any other lightweight note taking app with synchronization across devices works absolutely fine. Once I am on my desk, I move things into my notes directory as needed.
  2. Calendar events. I still believe that events belong into a traditional calendar app providing core features like reminders and recurring events. I use Google Calendar.

Conclusion

I know there are a million productivity tools, I tried most of them, and I also know that what I described is not rocket science distilled. It’s not even something new, people have been using similar approaches for years. For example, I find this Emacs Org-mode presentation from 12 years ago fascinating.

However, I do hope that this article will convince you to try a simple approach, or maybe combine 2-3 complimentary tools together to get the most out of your day. There is no need to force yourself to go all-in with an application that puts restrictions on you because it’s supposed to be the best productivity tool of the month.

Now you can probably see why Github Codespaces triggerred my brain. My favourite editor will be available across all my devices (laptop, mobile, tablet) through a web browser allowing me to edit my notes directly (assuming I use Github) from anywhere 🚀

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Battle of the Jamstack platforms — Netlify, Vercel, AWS]]> https://www.lambrospetrou.com/articles/battle-of-jamstack-platforms-netlify-vercel-aws/ battle-of-jamstack-platforms-netlify-vercel-aws Tue, 05 May 2020 00:00:00 GMT Overview

If you follow me online you probably know that I like serverless platforms a lot. Serverless nowadays is a pretty ambiguous term and can mean anything; from SQLite coining the term to AWS enumerating their serverless offerings.

In this article, serverless corresponds to a single page application (SPA) website that is accompanied by a lightweight API. This setup exploded in popularity recently, and it even got a name, Jamstack, for JavaScript, APIs, and Markup.

I experimented with many platforms over the past few years and I am going to briefly go over my current top 3 platforms to deploy your application, Netlify, Vercel, and AWS.

TL;DR

All three platforms are great and I recommend all of them, depending on how much time you are willing to spend, and how much extensibility you think you might need in the future.

Netlify

Website: https://netlify.com

I discovered Netlify couple of years ago, and have been tracking their progress ever since. The first time I saw it, I was simply stunned by how simple and fun it was to use while at the same time having powerful features.

Netlify has a lot of features and it feels more like an ecosystem of pluggable functionality (what they call add-ons), with flexible pricing.

Most notable add-on features:

  • Serverless Functions
  • Instant-forms
  • Identity
  • Analytics

The core product of the Netlify Platform is the combination of Netlify Build and Netlify Edge. Netlify Build is the ability to easily connect your Netlify project to your code repository (Github, Gitlab, BitBucket) and deploy your changes after every commit with a unique URL for each deployment. Netlify Edge is the application delivery network (ADN) which propagates the project’s artifacts in locations across the globe, similar to a normal content delivery network (CDN) but much smarter, and faster.

The whole process to get started is so simple that you can just drag-and-drop your project folder onto their website and deploy it in seconds!

I cannot possibly enumerate all the features Netlify provides but for the sake of this article we will focus on the core platform and the Serverless Functions add-on.

Serverless functions use AWS Lambda behind the scenes, but abstract it away so that we don’t have to fiddle with API Gateway, IAM role permissions, and all the nitty gritty AWS boilerplate.

For example, if we take JavaScript as an example, simply creating a file functions/hello-world.js with the content below will create an API accessible at /.netlify/functions/hello-world.

exports.handler = function(event, context, callback) {
  callback(null, {
    statusCode: 200,
    body: "Hello, World"
  });
}

That’s it for Netlify, let’s move on to the next one.

Vercel

Website: https://vercel.com

Vercel was until recently known as Zeit Now, and is extremely similar to Netlify in terms of target audience. Vercel however puts a lot of emphasis on their zero config deployments, and you can see it mentioned all over their website and docs. By zero-config deployment we mean that their system is trying to be smart and guess the build system or framework that your project is using based on your files and automatically do what it needs to do without you specifying anything. It works very well most of the time, apart from a small issue I discovered with the custom build system.

Vercel provides a similar experience as Netlify, where you can connect your repository and instantly build and deploy your project after every commit, and also includes a delivery network.

A big feature is once again their Serverless Functions offering, which is also using AWS Lambda under the covers. However, Vercel is a step up from Netlify, with more languages and more regions supported.

Its delivery network is also quite powerful, and more feature-rich than Netlify’s, since apart from the static assets, it can also cache serverless function responses.

Just for completeness, a serverless function example in JavaScript requires the following content in the file api/hello.js in order to expose the API at /api/hello?name=xxx.

module.exports = (req, res) => {
  const { name = 'World' } = req.query
  res.status(200).send(`Hello ${name}!`)
}

As you can see, even though it’s based on AWS Lambda, Vercel decided to use custom function signatures for the handler in contrary to Netlify which is using AWS’s format.

Before we finish with Vercel, I would like to briefly mention Next.js, the React framework they developed which is simply amazing. I recently migrated my blog to use this and I cannot emphasize how great it is, and in conjunction with Vercel’s platform they make a killer combination 🚀

Amazon Web Services (AWS)

Even though AWS does not provide a nice coherent Jamstack platform (I don’t like AWS Amplify at all), it provides all the necessary services to build your application.

For the past 5 years I have been using this myself, and the main services we need are Amazon S3 for storage of the static assets, Amazon CloudFront as our CDN, and AWS Lambda with API Gateway for our serverless functions API.

AWS has an advantage over both Netlify and Vercel, because of Lambda@Edge which is basically a slightly restricted version of AWS Lambda running on the edge locations of Amazon CloudFront, so much closer to the customers compared to the normal Lambda functions which are in the regional datacenters. I have been using Lambda@Edge for years now, both in personal projects but also while I was working at Amazon and I love it!

As you can see, it involves more moving pieces but they are all super robust services used by thousands of customers, serving billions of requests every year without issues. Some of the AWS services themselves are being built on-top of these services, which proves their reliability and that AWS bets on them working as expected!

Finally, I would strongly recommend using AWS CDK to provision your resources for all the above services in code, referred to as Infrastructure as Code (remember CloudFormation?).

Conclusion

I only scratched the surface of each platform’s feature set, however it’s quite clear that some are better at something and worse at something else.

If you focus only on Jamstack applications, my recommendation would be to go with Vercel go with Netlify (see update below). Recommending Next.js for one more time 😃

Update 2021-02-22: I have been experiencing slow load times (~2 seconds) on Vercel for two of my projects for a few weeks, and it seems that it’s happening when the page is rarely visited (although it shouldn’t according to docs, and Vercel support). Therefore, I now recommend Netlify over Vercel as a first choice. Netlify also works with Next.js even if you do not use the export mechanism, so feature parity is good.

If you are going to find uses for Netlify’s add-ons, then it’s a great choice as well! Really, Netlify and Vercel are very similar and you cannot go wrong with either. They even published an article yesterday on how to deploy a Next.js application on Netlify.

Finally, if your application is going to need additional cloud infrastructure to support it, like control over the AWS Lambda functions, queueing systems, databases, detailed monitoring, or anything else not provided by these Jamstack oriented platforms, then AWS is your best bet. You can always make requests from Netlify/Vercel’s serverless functions to other AWS resources, but it’s a matter of control and flexibility.

That’s it for this article, maybe I will do a deep-dive in the future for certain features…

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Hobby Languages for 2020]]> https://www.lambrospetrou.com/articles/hobby-languages-for-2020/ hobby-languages-for-2020 Sat, 25 Apr 2020 00:00:00 GMT Overview

Over the past few years I played around with many languages, starting with C/C++ at college to Java and JavaScript professionally, and to many others during my own time out of interest like Go, Scala, Python, Elixir, Clojure, Racket, Ruby and more.

During the last year I wanted to focus more on functional languages, specifically Elixir and Clojure. Unfortunately, I didn’t manage to do much due to work and other things taking up my time, but I really loved both languages.

After trying and playing with many languages I realised that there are certain things I am looking for in order to keep me interested for more than a few days.

My hobby language needs to:

  • Be fun to write
  • Have a good ecosystem to manage projects without a lot of ceremony
  • Be general purpose enough to write scripts and CLIs, but also web servers/clients
  • Be fast enough
  • Optional - Functional programming
  • Optional - Concurrency and Parallelism constructs

Based on the above requirements, this year I decided to focus on the following languages, and hopefully I will be able to spend a few months with each one and do some projects with them.

ReasonML

I had OCaml on my to-dive-deep list for years, but never jumped into it. When Facebook released ReasonML a while ago I thought it was a nice bridge/opportunity to get me into the OCaml world.

It’s no secret that I love the Node ecosystem, although it’s something most people hate these days. I truly believe that when you learn how to navigate around without getting disoriented by the huge amount of available modules, it provides a lot of things that other languages lack in a simple and easy way to use!

ReasonML has a fantastic integration with NPM which makes things easy.

The first project I am working on with ReasonML is to write an interpreter for the Monkey language, following the book Writing an Interpreter in Go by Thorsten Ball. I highly recommend this book, it’s simply awesome!

ClojureScript

Around 2 years ago I played around with Lisps, specifically Racket and Clojure.

Racket was surprisingly refreshing and I really enjoyed learning it, with great libraries and amazing documentation. However, it was quite slow for some things I did back then, but I know that some work was done to make it faster nowadays so I need to revisit it at some point.

Clojure, as a language trully impressed me. It has everything a functional language needs, with core and efficient datastructures, and ideas that are very powerful once learnt! However, although I have been using JVM languages professionally at work for 5+ years, I find the ecosystem to require too much ceremony to do the simplest thing, so I avoid it for my hobby projects.

ClojureScript on the other hand, is the same (almost) language as Clojure, but compiles to JavaScript. This means that you can use the NPM ecosystem to consume and produce software written in ClojureScript alongside JavaScript.

In addition, since Clojure has a very slow startup due to the way it handles class loading it’s not really suitable for quick scripts I write every now and then, whereas ClojureScript can use Node as its runtime leading to zero startup latency.

There are some amazing projects to make the ClojureScript integration with NPM easier, like Shadow CLJS which is my go-to; I even wrote a project template generator for it, create-shadow-cljs-app. Coincidentally, David Nollen also wrote an article yesterday about a new feature in ClojureScript to integrate better with JavaScript bundlers which will hopefully bring more people.

Clojure(Script) is so nice that the community wrote standalone REPLs to make it usable directly from the command line for scripting. My favourite is planck-repl, but babashka has been making the rounds recently as well and I will have to look into it.

I wrote a few small things in ClojureScript but I want to write something bigger to get a better feel of the language itself.

I will definitly have hard time deciding between ReasonML and ClojureScript since they both target Node, both can be used for similar things, and they are both functional languages. However, while they are very similar in some aspects, they are two very different languages, coming from different language families, Lisp vs ML.

Go

I started writing in Go back at 2014, and since then I wrote scripts, API servers, several static site generators for my website and other tools.

The language itself has nothing extraordinary, but it really feels nice using it and you can get many things done just by using the standard library, which is amazing on its own.

In addition, being a compiled language means it runs very fast, and it has excellent concurrency constructs that make writing concurrent software very easy, which is nice for the situations where you need to put those CPU cores into work.

I tried replacing Go with some other language over the past few years to be my quick go-to language but I always come back. Maybe ReasonML or ClojureScript will do the trick, but the simplicity, the performance, and the concurrency features are really unmatched so far.

Even though I already wrote several things in Go, I want to get back to it this year since I haven’t used it a lot recently.

Others

As I said in the overview, I really like playing around with many languages. Some of them lose me from the hello world, but many of them end up in my to-dive-deep list.

Other languages I really liked and I would like to spend more time with them, probably not a lot this year, are Elixir and Rust.

Elixir has everything I love in a language apart from single-threaded performance really. It’s amazing for anything that requires a lot of juggling around since it supports millions of processes running independently, with their own garbage collection, preemptive scheduling, and other great features provided by its runtime, the BEAM VM. As a language, and as an ecosystem it is very high on my list! There is some amazing work being done now with Phoenix LiveViews which is very interesting, and I am eager to see where it will lead.

Rust is making waves recently a lot! Both in my social circles, but also professionally it gets adopted by many big companies for their performance-critical systems. It has a nice combination of functional programming concepts together with performance oriented features like memory management to avoid runtime issues. In addition, it is among the best languages to use in order to compile to WebAssembly which is something I am extremely interested in.

I wish the days were longer :) Much to learn, so little time…

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Amazon Leadership Principles — Choose 3]]> https://www.lambrospetrou.com/articles/amazon-leadership-principles/ amazon-leadership-principles Sat, 18 Apr 2020 00:00:00 GMT Overview

After almost 5 years at Amazon, I really think that the Amazon Leadership Principles are among the few things that are always going to be engraved in me.

In the yearly reviews (so-called Forte) we choose up to 3 LPs for some of our colleagues to describe their superpowers and their growth opportunities, so in a similar fashion I will enumerate the 3 LPs that I consider to be the most crucial, and which have been the barometer to everything I was doing.

Customer Obsession

Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.

Not much to say… Customers come first, so the customer should be the sole focus of anything you do. Choosing the right problems to solve, making the right decision between trade-offs, and delivering the final solution should all start and finish with the customer in mind.

Insist on the Highest Standards

Leaders have relentlessly high standards — many people may think these standards are unreasonably high. Leaders are continually raising the bar and driving their teams to deliver high quality products, services, and processes. Leaders ensure that defects do not get sent down the line and that problems are fixed so they stay fixed.

I cannot emphasize this enough. Keeping the bar high, and always insisting on high standards is not only important for the short-term but it’s absolutely crucial for the long-term.

We face the temptation of doing something quick-and-dirty pretty much in every line of code we write or decision we make. Choosing this path enough times though means that the product will degrade over time, but even worse, we make the job of our future self or future colleagues super hard.

How many times did you want to have two words with the author of some part of a system you worked on… :)

Deliver Results

Leaders focus on the key inputs for their business and deliver them with the right quality and in a timely fashion. Despite setbacks, they rise to the occasion and never compromise.

This is the culmination of all the Leadership Principles together in my opinion. You have to focus on the key inputs for your business, and deliver them with the right quality, and in a timely fashion, while at the same time never compromising.

Yeah, I pretty much re-stated the same thing, but this is the gist of it. At the end of the day, you have to deliver something to your customers, but the difficulty lies in doing it in a timely fashion without compromising quality.

Conclusion

I honestly think that all 14 (as of now) LPs are super important!

Having said that, it’s almost impossible for every individual to have the same definition and the same bar for all 14 of them, and this is something I have seen a lot over the past few years. Each person will interpret each LP in their own way, which is fair and expected.

The magic though happens when a team does not lose focus on the Leadership Principles and always comes back and re-evaluates everything against them!

I am glad I had the chance to be an Amazonian!

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Meiosis pattern - Unidirectional data flow web applications using streams]]> https://www.lambrospetrou.com/articles/meiosisjs-typescript-mithril-demo/ meiosisjs-typescript-mithril-demo Fri, 14 Sep 2018 00:00:00 GMT Problem

After using ReactJS for two years, along with all its ecosystem (Redux for state management, Redux-saga for managing side-effects, etc.) I wanted to experiment with a simpler approach where I was not forced to use many framework specific libraries, but at the same time having the same uni-directional flow of data and the ease of UI creation using components.

A few months ago, I came across Mithril, a lightweight component based framework which uses pure Javascript objects for its components. I was instantly hooked and wanted to use it in all my personal projects.

How can I get the state management I want without being tied to a framework so that I can use Mithril or React interchangeably without issues?

TL;DR

My answer is using Meiosis pattern.

Streams

In order to properly understand the solution let’s go over some basics regarding streams.

There are several Javascript stream libraries but for the rest of the article I will use Mithril/stream.

Meiosis requires three basic operations on streams:

  1. Updating the stream value
  2. Mapping over the stream in order to listen for updates to the stream’s value
  3. Scanning over the stream in order to calculate accumulated value after every update

Updating the stream value

import * as ms from 'mithril/stream'

const s = ms()
s(1)
console.log(s())  // 1
s(s()+10)
console.log(s())  // 11

The example above shows that just calling a stream as a function will return the latest value of the stream, and supplying a value to the function application will update the stream’s value.

Also, the fact that the stream itself is a function can be super useful since it can be composed into more complex functions.

Stream.map

import * as ms from 'mithril/stream'

const s = ms()

s.map(value => console.log(`updated value: ${value}`))

s(1)    // prints "updated value: 1"
s(20)   // prints "updated value: 20"
s(311)  // prints "updated value: 311"

The Stream.map operation is simple. It just executes a given function every time the stream value is updated, and also returns a new stream. The return value of the given function after every execution becomes the new value for the stream returned by the call to map().

Stream.scan

import * as ms from 'mithril/stream'

const accumulate = (accumulator, value) => accumulator * value
const processValue = value => console.log(value)

const s = ms()
const scanned = ms.scan(accumulate, 1, s)
scanned.map(processValue)

s(1)    // prints 1
s(2)    // prints 2
s(5)    // prints 10
s(100)  // prints 1000
s(0.5)  // prints 500

The distilled functionality of Stream.scan is that whenever the value of the source stream (s) changes, the accumulate function is called being passed the current accumulator value and the new source stream value. The return value will become the new value of the scanned stream and the accumulator value being passed to the next accumulate invocation.

The fascinating idea here is that a stream can have functions as its values.

In the following example we pass functions as values and as a result we can apply different operations on the accumulator value each time.

import * as ms from 'mithril/stream'

const update = (accumulator, fn) => fn(accumulator)
const processValue = value => console.log(value)

const s = ms()
const scanned = ms.scan(update, 1, s)
scanned.map(processValue)

s(value => value + 1)         // prints 2
s(value => value - 1)         // prints 1
s(value => value * 10)        // prints 10
s(value => 1234)              // prints 1234
s(value => value / 2)         // prints 617

Spend a few minutes to fully understand the above example because it’s the bread and butter of the Meiosis pattern.

Notice how we pass functions in the stream and by the use of scan they are applied in order transforming the stream accumulated value.

Meiosis Pattern

The Meiosis website is doing an amazing job describing the pattern and includes a brief video showcasing how to use the pattern.

Meiosis was inspired by similar projects striving for unidirectional flow of data to make state management easy like SAM Pattern, Elm Architecture, CycleJS, and others.

The whole Meiosis pattern is contained in the following code:

import * as ms from 'mithril/stream'

class AppModel {
  counter: number = 0
}
type ModelUpdateFunction = (model: AppModel) => AppModel
type UpdateStream = ms.Stream<ModelUpdateFunction>

const createApp = (update: UpdateStream) : MeiosisApp => ({ 
  initialModel: ..., view: ..., render: ...
})

const setupMeiosis = (
  createApp: (s: UpdateStream) => MeiosisApp,
  container: Element
) => {
  const update = ms<ModelUpdateFunction>()
  const modelUpdate = (model: AppModel, fn: ModelUpdateFunction) => fn(model)

  const app = createApp(update)

  const models = ms.scan(modelUpdate, app.initialModel(), update)
  models.map(model => app.render(container, app.view(model)))
}

setupMeiosis(createApp, document.body.querySelector('#app') as Element)

Let’s examine each section separately to fully understand the pattern.

const update = ms<ModelUpdateFunction>()

The code above creates the update stream that has functions as its values. These functions receive an AppModel and return a new AppModel. These functions are going to be our actions, that are triggered either by a user interaction or by an asynchronous task, fetch, etc.

const app = createApp(update)

This creates our application, which receives the update stream that will be used by the application’s actions.

In addition, the app exposes three methods:

  • initialModel() returns the initial AppModel which will act as the bootstrapping model for the application.
  • view(model) receives the latest AppModel and returns the view to be rendered. This view can be of any framework (e.g. React, Mithril).
  • render(element, view) receives the application view, and the DOM element to which the view will be rendered. Meiosis does not care about the framework used as long as there is functionality to render a view on-demand. Mithril, React, Preact, Inferno, etc. are able to do this efficiently with the use of virtual dom.
const models = ms.scan(modelUpdate, app.initialModel(), update)
models.map(model => app.render(container, app.view(model)))

This is the juicy part of the pattern.

The models stream is a scanned stream over the update stream. This means that any function passed into the update stream, due to the scan functionality described previously, will be used to transform the current application model into a new model, and then set that model as value in the models stream.

The second line applies a map operation over the models stream. This will render the application view using the updated application model every time there is a new value pushed to the update stream. Remember that the values of the update stream are functions that transform the model.

This deviates slightly from traditional frameworks in the sense that when an action occurs, instead of publishing the updated model itself, the transformation functions are pushed, and the Stream.scan functionality takes care of applying those transformations to the model/state.

Demo

The following code is a complete demo implementing a counter, with plus/minus buttons.

import * as m from 'mithril'
import * as ms from 'mithril/stream'

class AppModel {
  counter: number = 0
}
type ModelUpdateFunction = (model: AppModel) => AppModel
type UpdateStream = ms.Stream<ModelUpdateFunction>

interface MeiosisApp {
  model: () => AppModel
  view: (model: AppModel) => m.Vnode
  render: (el: Element, v: m.Vnode) => void
}

const createActions = (update: UpdateStream) => ({
  inc: (value: number) => update(model => ({counter: model.counter + value}))
})

const createApp = (update: UpdateStream): MeiosisApp => {
  const actions = createActions(update)
  return {
    model: () => ({counter: 0}),
    view: (model: AppModel) => m("div", [
      m('p', model.counter),
      m('button', {onclick: () => actions.inc(1)}, 'plus'),
      m('button', {onclick: () => actions.inc(-1)}, 'minus')
    ]),
    render: (container: Element, v: m.Vnode) => m.render(container, v)
  }
}

const setupMeiosis = (
  createApp: (s: UpdateStream) => MeiosisApp,
  container: Element
) => {
  const update = ms<ModelUpdateFunction>()
  const modelUpdate = (model: AppModel, fn: ModelUpdateFunction) => fn(model)

  const app = createApp(update)

  const models = ms.scan(modelUpdate, app.model(), update)
  models.map(model => app.render(container, app.view(model)))
}

setupMeiosis(createApp, document.body.querySelector('#app') as Element)

Demo repository on Github

Conclusion

After working with most of the Javascript frameworks the last few years, I enjoy having the following while developing an application:

  • Immutability of state/data
  • Unidirectional flow of data (actions => update state => render updated view => actions)
  • Fast component rendering (the best solutions as of now use virtual dom diffing)

Adopting Meiosis pattern in my applications, means that all of the above are possible, and the only thing tied to a specific framework is really just the view components. Taking the above code, and replacing Mithril with React is just a matter of a few minutes.

I hope that more engineers will start adopting patterns that are applicable to all kinds of Javascript applications instead of focusing on framework specific solutions.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Run multiple services on a single EC2 instance using AWS Elastic Beanstalk (Go and Multicontainer Docker platforms)]]> https://www.lambrospetrou.com/articles/multiple-services-elastic-beanstalk/ multiple-services-elastic-beanstalk Sun, 11 Mar 2018 00:00:00 GMT Problem

Many times I want to run multiple services on the same EC2 instance. Sometimes I am doing a toy project and I don’t want to pay for resources just hosting each project on its own, and other times, even in production systems, I need to deploy multiple microservices on the same instance, e.g. Nginx proxy, the application web service, and maybe some other monitoring service.

I am a huge fan of AWS Elastic Beanstalk and I will explain how we can achieve multi-service on same instance setup using the Go Platform and the more flexible Multicontainer Docker Platform.

Desired result

The example application I will deploy has two Go web services running which just serve some static HTML and an Nginx proxy doing the routing between them.

Web service 2 handles any request under the /web2 path, and Web Service 1 handles everything else.

> curl http://multiplegoservices-env.mzkfjw36fh.eu-west-1.elasticbeanstalk.com/web2
Service 2 Path, "/web2"

> curl http://multiplegoservices-env.mzkfjw36fh.eu-west-1.elasticbeanstalk.com/web1
Service 1 Path, "/web1"

> curl http://multiplegoservices-env.mzkfjw36fh.eu-west-1.elasticbeanstalk.com/
Service 1 Path, "/"

The code for our services is exactly the same between the two platforms, and the only difference is among the files specific for each platform configuration.

Go Platform

The Go Platform in Elastic Beanstalk is pretty simple, with the most important concepts being the Procfile, the Buildfile, and the special .ebextensions folder which we will use to provide custom configuration to the Nginx proxy deployed automatically on the instances.

Source code available on Github.

Procfile

This file contains a simple enumeration of the services to start (i.e. executables to run). For the example application, Procfile contains the following:

web_service1: bin/web-service-1
web_service2: bin/web-service-2

One thing you need to know is that Elastic Beanstalk will start the first service setting the environment variable PORT=5000, and that is the port your service should listen for requests. Each subsequent service will receive PORT values in 100 increments from the last one, i.e. web-service-2 will have PORT=5100.

Buildfile

This file contains a simple enumeration of commands to run during the deployment artifact build time. This can be used to build your code and generate the executables that Procfile will execute, but in my case I like building locally (or in a pipeline) and just deploy the executables as the Elastic Beanstalk artifact.

Just for the sake of using Buildfile, I call a bash script that prints Hello world!, and it looks like below:

command_to_run_during_build: bin/hello.sh

As a general guideline, the Buildfile can be used for any arbitrary task that needs to run before the services are started.

Nginx proxy configuration

For the proxy configuration we just need a server {} Nginx directive to provide the routing between the two services. To achieve this, we create a .conf file inside the .ebextensions/nginx/conf.d/ directory which will be included by Nginx during startup.

The following configuration is enough to do the job, and is in the file .ebextensions/nginx/conf.d/01_proxy.conf.

server {
    server_name .elasticbeanstalk.com;
    listen 80;

    location /web2 {
        proxy_pass http://127.0.0.1:5100;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Deployment artifact

All the paths used in the files above are based on the following artifact (the .zip file) I deploy to Elastic Beanstalk. Full listing of the contents below:

> unzip -l build/bundle.zip 
Archive:  build/bundle.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       64  2018-03-11 13:44   Procfile
        0  2018-03-11 13:44   .ebextensions/
        0  2018-03-11 13:44   .ebextensions/nginx/
        0  2018-03-11 13:44   .ebextensions/nginx/conf.d/
      319  2018-03-11 13:44   .ebextensions/nginx/conf.d/01_proxy.conf
        0  2018-03-11 13:44   bin/
  6218916  2018-03-11 13:44   bin/web-service-1
       38  2018-03-11 13:44   bin/hello.sh
  6218916  2018-03-11 13:44   bin/web-service-2
       42  2018-03-11 13:44   Buildfile
---------                     -------
 12438295                     10 files

Multicontainer Docker Platform

The multicontainer Docker platform uses Amazon Elastic Container Service under the covers, but as I said before, deploying through Elastic Beanstalk makes things a lot easier!

There is only one important configuration file in the Multicontainer Docker platform and that is the Dockerrun.aws.json.

This file contains Docker specific definitions, e.g. the Docker images we want for each service, the volume definitions mapping to source paths in our deployment artifact, etc.

Source code available on Github.

Dockerrun.aws.json

In order to achieve the same result as with the Go Platform I use the following Dockerrun.aws.json file.

{
  "AWSEBDockerrunVersion": 2,
  "volumes": [
    {
      "name": "web1",
      "host": {
        "sourcePath": "/var/app/current/web-service-1"
      }
    },
    {
      "name": "web2",
      "host": {
        "sourcePath": "/var/app/current/web-service-2"
      }
    },
    {
      "name": "nginx-proxy-conf",
      "host": {
        "sourcePath": "/var/app/current/proxy/conf.d"
      }
    }
  ],
  "containerDefinitions": [
    {
      "name": "web1",
      "image": "golang:1.10",
      "essential": true,
      "memory": 128,
      "mountPoints": [
        {
          "sourceVolume": "web1",
          "containerPath": "/var/app"
        }
      ],
      "portMappings": [
        {
          "hostPort": 5000,
          "containerPort": 5000
        }
      ],
      "environment": [
        {
          "name": "PORT",
          "value": "5000"
        }
      ],
      "command": ["/var/app/web-service-1"]
    },
    {
      "name": "web2",
      "image": "golang:1.10",
      "essential": true,
      "memory": 128,
      "mountPoints": [
        {
          "sourceVolume": "web2",
          "containerPath": "/var/app"
        }
      ],
      "portMappings": [
        {
          "hostPort": 5100,
          "containerPort": 5100
        }
      ],
      "environment": [
        {
          "name": "PORT",
          "value": "5100"
        }
      ],
      "command": ["/var/app/web-service-2"]
    },
    {
      "name": "nginx-proxy",
      "image": "nginx",
      "essential": true,
      "memory": 128,
      "portMappings": [
        {
          "hostPort": 80,
          "containerPort": 80
        }
      ],
      "links": [
        "web1", "web2"
      ],
      "mountPoints": [
        {
          "sourceVolume": "nginx-proxy-conf",
          "containerPath": "/etc/nginx/conf.d"
        },
        {
          "sourceVolume": "awseb-logs-nginx-proxy",
          "containerPath": "/var/log/nginx"
        }
      ]
    }
  ]
}

Notes

  • /var/app/current is the directory on the host machine that contains our deployment artifact, i.e. the .zip file unzipped.

  • In order to allow the Nginx image to communicate with the two services running we need to link those images to the nginx-proxy image, and instead of using http://127.0.0.1 in the Nginx .conf file we should use http://web1 and http://web2 as shown below.

    server {
    server_name .elasticbeanstalk.com;
    listen 80;

    location /web2 {
        proxy_pass http://web2:5100;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location / {
        proxy_pass http://web1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Deployment artifact

Full listing of the deployment artifact contents is as follows:

> unzip -l build/bundle.zip 
Archive:  build/bundle.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     1960  2018-03-10 19:01   Dockerrun.aws.json
        0  2018-03-10 19:01   proxy/
        0  2018-03-10 19:01   proxy/conf.d/
      326  2018-03-10 19:01   proxy/conf.d/default.conf
        0  2018-03-10 19:01   web-service-1/
  6218916  2018-03-10 19:01   web-service-1/web-service-1
        0  2018-03-10 19:01   web-service-2/
  6218916  2018-03-10 19:01   web-service-2/web-service-2
---------                     -------
 12440118                     8 files

Conclusion

AWS Elastic Beanstalk is an amazing service which abstracts a lot of stuff that are not really part of the application, e.g. load balancers, autoscaling groups, logging, alarms, and there is even a super helpful dashboard right out-of-the box.

My simplistic guideline on what platform to use is as follows:

Go Platform

  • Your services are written in Go
  • Your services can be compiled into binary executables that run on Amazon Linux

Multicontainer Docker Platform

  • Anything else

Have fun microservicing with AWS Elastic Beanstalk!

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Encrypt files with password on Linux]]> https://www.lambrospetrou.com/articles/encrypt-files-with-password-linux/ encrypt-files-with-password-linux Sat, 06 Jan 2018 00:00:00 GMT Problem

I have some important private files that I want to store in Google Drive and on my USB flash drive, but I don’t want them to be in plain sight for anyone to see.

I would like to at least password-protect them before storing them, but without too much hassle with asymmetric cryptography where I need to fiddle with keys.

Solution

It turns out pretty much all UNIX systems have GnuPG installed which allows me to just run a command to encrypt a file using a passphrase, and a corresponding command to decrypt it when I need to open it.

I found out that this method is also used inside NASA when transferring files.

In order to encrypt and password-protect a file run the following command:

gpg -c --cipher-algo AES256 private-file.txt

The -c option specifies that we want to do symmetric encryption using a passphrase. The --cipher-algo 256 option specifies that we want to use the AES256 cipher instead of the default CAST5 cipher, although this is not required.

The above command will ask you for the passphrase to use, and then will create a new file named private-file.txt.gpg, which is the encrypted and password-protected file we want to store.

In order to decrypt the file run the following command:

gpg private-file.txt.gpg

Once you enter the passphrase used during the encryption of the file, you will get back the decrypted file which will have the same name without the .gpg extension, hence private-file.txt.

Tips

  • If you want to encrypt a whole directory (folder), then you have to first zip/tar the folder into a single file and then apply the same command above to the zipped/tarred file.
  • Use long passphrases for important files consisting of multiple words with letters, spaces, symbols, and numbers to maximise the entropy and security of the encryption.

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Export environment variables from multiple files using Bash on Linux]]> https://www.lambrospetrou.com/articles/export-environment-variables-multiple-files-linux-bash/ export-environment-variables-multiple-files-linux-bash Sun, 19 Nov 2017 00:00:00 GMT Problem

Recently, I have updated the deployment scripts of my website and for some tasks I wanted to have certain environment variables available in my scripts. For example, I wanted to invalidate the Cloudfront distribution so that I don’t have to wait for the cache to expire before serving the new content, and I needed the --distribution-id.

I looked for the easiest way to export environment variables from key-value pairs in a file, and I was delighted to see that it can done with a single line!

Solution

The solution is based on this StackOverflow answer.

Assuming the following aws.environment file:

DISTRIBUTION_ID=xxxxxxxxx
S3_BUCKET=www.example.com

The following script will export the above key-value pairs as environment variables.

#!/usr/bin/env bash

source <(sed -E -n 's/[^#]+/export &/ p' aws.environment)

# ... Commands that use the above variables i.e. echo "$DISTRIBUTION_ID"

In addition, I didn’t want to limit myself only to one file of environment variables because I would like to version control some of them in git. So, I wanted to scan and export variables from all the files in a given directory. This can be done using the following script.

#!/usr/bin/env bash

source <(find ./build-tool/env/ -type f -exec sed -E -n 's/[^#]+/export &/ p' {} +)

# ... Commands that use the above variables

This final adaptation uses the find command to read all files in the ./build-tool/env directory and for each one exports its key-value pairs as environment variables.

That’s it! Once again, knowledge of some nifty Linux commands saves the day.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[npm package to easily run arbitrary compiled binaries in your applications or AWS Lambda]]> https://www.lambrospetrou.com/articles/aws-lambda-binary-npm-helper/ aws-lambda-binary-npm-helper Wed, 18 Oct 2017 00:00:00 GMT I recently posted a detailed article on how to efficiently run Racket compiled native binaries on AWS Lambda, and the same process can be used for any compiled binary that runs on Amazon Linux (including binaries from languages like Go, Racket, OCaml, Rust, C/C++, etc.).

The Lambda wrapper code was very easy to understand but a bit long, and if you have lots of lambdas using binaries you might end up copy-pasting those lines, which as we know is not good. Remember the DRY principle?

As a result, to make it easier for me, and anyone implementing the solution described in that article, I created an npm package to help you reduce the boilerplate of your Lambda wrapper code significantly.

AWS-Lambda-Binary npm package

You can find the package at https://www.npmjs.com/package/aws-lambda-binary

Its usage is very easy and the package itself provides sufficient documentation of the API. In addition, couple of ready-to-upload AWS Lambda examples are provided inside the _examples folder.

In a nutshell, the following describes how to use this package to start the Linux command cat which will be used as an echo-back process.

First of all if you don’t have an existing lambda project, just create one using the following commands:

mkdir lambda-test && cd lambda-test
npm init -y

Now that you have a project, install the npm package:

npm install aws-lambda-binary

Copy and paste the following code into a file named wrapper.js.

const application = require('aws-lambda-binary').spawnLineByLine({
    spawn: { command: 'cat' }
});

exports.handler = function (event, context) {
    application.ensureIsRunning();

    application.stdout(result => context.done(null, result));

    application.stdin(JSON.stringify({event, context}));
};

Finally, prepare the zip file to upload to AWS Lambda by running the command below:

zip -r bundle.zip wrapper.js node_modules/

In the AWS Lambda console, create a new function with Runtime Node.js 6.10 (or higher), and with Handler wrapper.handler.

Upload the bundle.zip file you created above and Save and Test. The above lamba code takes the event of the lambda invocation, along with the context object which contains some metadata, serialises them into JSON string and sends them to the cat process through standard input. The process returns the result back through standard output and we successfully finish the lambda invocation passing the line of text received as the result.

There are more examples in the npm package but as you can see above, once you have all your logic into a compiled binary, starting the binary and communicating with it is a matter of ~5 lines.

Enjoy, and feel free to contribute to the package.

SEO tags: AWS, AWSLambda, binary, racket, go, ocaml, rust

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Command prompt alternatives for Windows]]> https://www.lambrospetrou.com/articles/windows-command-prompt-alternatives/ windows-command-prompt-alternatives Mon, 16 Oct 2017 00:00:00 GMT Since I have been fiddling around with Windows Subsystem for Linux I realised how bad Command Prompt is compared to Terminator, my go-to terminal solution for Linux.

I wanted to find an alternative which at least offered multiple tabs, proper text selection and in general being a better terminal for Windows.

I tried several ones but for now I settled with ConEmu which has the extra benefit of being open-source.

One problem I had with it, is that when you go into the WSL mode, using the bash command, the arrow keys do not work. This was weird and confusing at first since they work if you go into WSL mode using Command Prompt.

The solution is to pass an additional argument when entering the WSL mode through ConEmu, as shown below.

# `p1` refers to tab-1.
# If you are entering this command to a different tab in ConEmu
# you have to use the proper tab-id.
bash -cur_console:p1

# Or you can open a new tab straight into WSL mode and avoid the tab-id
bash -new_console

Now all the keys should work properly.

This nice terminal application, along with the ability to use editors straight from the Linux subsystem as described in a previous article make Windows 10 a very viable solution for developers that love the command line.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Windows Subsystem for Linux - Editor (GUI) Setup]]> https://www.lambrospetrou.com/articles/windows-linux-subsystem-editor-setup/ windows-linux-subsystem-editor-setup Sun, 15 Oct 2017 00:00:00 GMT Problem

While I was testing out the Windows Subsystem for Linux - WSL I setup all my favourite programming languages inside the Linux system instead of Windows. Everything was working smoothly until I had to use an editor to work on some projects.

I initially installed Visual Studio Code in my Windows system but realised that I don’t have access to the PATH or the binaries of my languages installed inside WSL. The solution is to have an X-server running in Windows and then run the editor from inside the WSL but rendering to the X-server display (easier than it sounds).

BTW, if you are a VIM user you can stop reading right now since VIM works out of the box.

Solution

Firstly, we need to install an X-server in Windows. I installed Xming X Server for Windows and it works like a charm, with all the default options. Just download, install, and run!

If the server is running successfully, you should see an icon in the notification area of the taskbar that says Xming Server:0.0.

Once the X-server is running we are ready to go. In order to start an application with a GUI from inside WSL and have it render in Windows we need to set the environment variable DISPLAY=:0 and then run the command.

# Either set it right before the command
DISPLAY=:0 command-with-a-gui

# or set it for the whole terminal session
export DISPLAY=:0
command-with-a-gui

Visual Studio Code

I started using Visual Studio Code for the most part of last year and I was really happy with it, but it seems that it cannot properly run through an X server and it is not a priority of its team to get it working.

Not a huge issue since my previously favourite editor is perfectly working, but still a bummer since Microsoft’s own editor does not work with their system!

Sublime Text 3

The Sublime Text 3 installation instructions are straightforward but I copy the commands needed here to make the article more complete.

wget -qO - https://download.sublimetext.com/sublimehq-pub.gpg | sudo apt-key add -
echo "deb https://download.sublimetext.com/ apt/stable/" | sudo tee /etc/apt/sources.list.d/sublime-text.list
sudo apt-get update && sudo apt-get install sublime-text

The command to run Sublime Text 3 is subl. So in conjunction with the X-server display variable we can run Sublime using the following command:

DISPLAY=:0 subl

The above command should open a window of Sublime Text inside Windows, and if you try to Open folder you should be seeing the Linux subsystem file-system.

The next step is to install the package control for Sublime so that we can install our favourite plugins.

Next steps

The font I use for all my text editors, and IDEs, is Source Code Pro, which in Ubuntu-like systems can be installed easily using the following commands:

git clone --depth 1 --branch release https://github.com/adobe-fonts/source-code-pro.git ~/.fonts/adobe-fonts/source-code-pro
fc-cache -f -v ~/.fonts/adobe-fonts/source-code-pro

In addition, there are some Sublime options that is good to change from the start. Below you can see the changes I did to my User settings file (Preferences -> Settings).

{
    "font_face": "Source Code Pro",
    "font_size": 10,
    "spell_check": true,
       "always_show_minimap_viewport": true,
       "highlight_line": true,

       "trim_trailing_white_space_on_save": true,
       "show_encoding": true,
       "show_line_endings": true,

       "dictionary": "Packages/Language - English/en_GB.dic",
}

Conclusion

I am really impressed by the work Microsoft did with the Windows Subsystem for Linux. I am a Linux user for almost a decade but I always had those days that I had to access a Windows system, either for Microsoft Office, or a BIOS update, or something else. As a result, I always setup my machines to dual-boot Windows and Linux.

With WSL, I have a good feeling that it could be a nice alternative to dual boot. Apart from some limitations with GUI applications not running through an X-server, I didn’t have any problem running my favourite languages and tools in WSL. Including Node.js, Racket, Go, Elixir and Erlang, and of course native Linux tools like SSH.

In conclusion, nicely done by Microsoft, and I really hope that this will continue to improve and get polished in order to reach the stability levels we require for day-to-day usage. This could be what Microsoft needed to make Windows a viable option for developers loving their Linux!

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[AWS Lambda meets Racket (or any compiled language)]]> https://www.lambrospetrou.com/articles/aws-lambda-meets-racket/ aws-lambda-meets-racket Wed, 11 Oct 2017 00:00:00 GMT Problem

I recently started learning Racket and one of the first things I do with a new language is integrating it with AWS Lambda.

One of my favourite languages is Go, and since both Racket and Go can be compiled down to self-contained executable binaries we can re-use some of the knowledge running Go on AWS Lambda.

AWS Lambda Overview

Before delving into the detailed solution that works best for me I will provide the different ways we can run code in Lambda.

Language natively supported by AWS Lambda

If your language is supported natively by Lambda then it’s very easy and you should follow the AWS documentation for the language. As of the time of writing, the supported languages include Node.js, Java, Python, and .NET Core.

Language can be compiled to C shared library

If your language of choice can be compiled down to a shared C-library binary, then the most performant way to run on Lambda, is to use a wrapper in Python that loads this library and directly make calls to your shared library.

This is the best way I have found to run Go code so far and I make use of the great library eawsy/aws-lambda-go (For Go 1.8+ there is a new version which uses Go plugins).

This approach can be used to run any language compiled down to a C shared library.

Language can be compiled to standalone binary that runs on Amazon Linux

If your language of choice cannot be compiled to shared library, but can be compiled to standalone binary that can run on the Amazon Linux system (which is the OS AWS Lambda uses behind the scenes) then it’s very easy to run the application binary through Lambda.

The easiest and simplest way to run your code is to spawn a new subprocess of your binary using some lambda wrapper code (in Python or Node.js) and pass the event and/or context as input to that process. Then, you read the standard output of the subprocess from inside the wrapper code and return it to the caller of your Lambda code.

The only downside of this approach is that each Lambda function invocation is pretty slow, since a new process has to be started every time your function runs.

For example, using the code from Daniel’s Brunner blog post each Lambda invocation configured with 128MB memory has an average runtime of ~450-550ms. You can boost the performance of this if you increase your Lambda’s memory to the maximum of 1536MB. This will bring the average runtime down to ~50-150ms.

The reason behind this interesting fact is that increasing your Lambda function’s memory, you also increase its CPU power, hence leading to much faster subprocess spawning times.

This spawning per function invocation works fine, but it’s still slow for me and I don’t want to pay for the maximum Lambda memory to get good performance. My proposed solution is a very simple adaptation of this approach.

Since Lambda has a static initialisation section every time the underlying container is started, I will spawn a process during that time, and then in each function invocation I will communicate over standard input and output with the subprocess from inside the lambda wrapper code.

This has a tremendous speedup over creating a new process every time since the average invocation runtime for the 128MB using this approach is 1-100ms, and consistently stays under ~10ms with the 1536MB memory configuration. This speedup is significant for lambdas that run many times over a period of time because the overhead of spawning a process is only observed the first time our code will run in a specific container instance, and then the same process is re-used leading to these extremely fast times.

See proof below, using the 128MB memory configuration, and runtime of 0.51ms (yes that’s less than a millisecond)!

AWS Lambda running Racket

Solution

As explained in the previous section, my best solution so far which keeps the complexity to a minimum, is to spawn a subprocess of our Racket application binary during the static initialisation of the Lambda function and re-use that process during the individual lambda invocations, by communicating over stdio.

Racket application

I will use the following Racket code as example, which just echoes back the input of the application.

#lang racket/base

;; This is the actual logic of our code!
(define (execute-logic data)
  (display (format "data: ~a~%" data))
  (flush-output))

;; The following code waits for one line of input and then dispatches it to the `execute-logic` function.
;; This way we can have full control over what we can do and there can be an arbitrarily complex protocol
;; between the caller and this application over **stdio**.
(define (loopInput)
  (execute-logic (read-line))
  (loopInput))
(loopInput)

The last four lines in the code above is just a loop that reads a line from standard input, and then calls our execute-logic function passing the data received. This is the only boilerplate needed by our application code.

Your logic can do whatever it wants with the input data and then just print the result to standard output. Here we just use the display function to print the input to standard output.

Wrapper code in Node.js

Update@2017-10-18: I created an npm package to significantly reduce the boilerplate code shown below, so after you have read this article and understood how the solution works, use the package AWS-lambda-binary in your production Lambda functions.

The wrapper code will be a bit longer but still remains very simple to understand.

const child_process = require('child_process');
const readline = require('readline');

/****************************
 * START OF WRAPPER CODE
 ****************************/
const execPath = './application';

let proc = initProc();

function initProc(options) {
    options = options || {};
    const p = child_process.spawn(execPath);

    // Add your own custom handler if you want to handle the errors differently.
    p.stderr.on('data', (err) => {
        console.error('proc stderr: ', err);
    });

    // You might want to use ```exit``` event instead of ```close``` if you don't
    // care about the ```stdio streams``` of the subprocess.
    // https://nodejs.org/api/child_process.html#child_process_event_close
    // https://nodejs.org/api/child_process.html#child_process_event_exit
    p.on('close', function (code) {
        if (code !== 0) {
            console.error(new Error(`Process closed with code: ${code}`));
        }
        const {handlerProcCloseCallback} = proc;
        proc = null;
        if (handlerProcCloseCallback) {
            handlerProcCloseCallback(code);
        }
    });

    // This is the part that you get the result back from your Racket application
    // I prefer to receive lines back from the application for simplicity
    // so I use https://nodejs.org/api/readline.html#readline_event_line
    // but you can adapt this to use binary data as well, exactly like we did with
    // `stderr` above.
    const rl = readline.createInterface({ input: p.stdout });
    rl.on('line', (line) => {
        const {handlerCallback} = proc;
        if (handlerCallback) {
            handlerCallback(line);
        }
    });

    return {
        p, rl,

        // Will be called for every **line** output from our application.
        handlerCallback: null,

        // Will be called when the application process is closed. You can use
        // this callback to restart it automatically or do something custom.
        handlerProcCloseCallback: null,
    };
}

function ensureProcRuns(options) {
    if (!proc) {
        proc = initProc(options);
    }
    if (options.resetCallbacks) {
        proc.handlerCallback = null;
        proc.handlerProcCloseCallback = null;
    }
}
/**************************
 * END OF WRAPPER CODE
 **************************/

exports.handler = function (event, context) {
    ensureProcRuns({resetCallbacks: true});

    // Register the handler we want for each line response!
    proc.handlerCallback = (result) => {
        console.log(`rkt: ${result}`);
        context.done(null, `result: ${result}`);
    };

    // Send the input to the Racket application
    proc.p.stdin.write(`${JSON.stringify({event, context})}\n`);
};

I don’t think it needs lots of explanation but I will explain some of the important bits.

Your main focus should be inside the exports.handler = function(...) {...} section, and everything before that is just the wrapper code around the subprocess spawning.

The first thing we need to do inside our handler code is to call ensureProcRuns() in order to make sure that there is a running subprocess of our Racket application. Ideally, this should always be instant since we already instantiate a process during the initialisation phase of the container. If the subprocess fails though and exits during an invocation, this will ensure that future invocations will re-spawn the process.

After that first line, we need to register a callback function that will handle the result back from the Racket application. In my case here we just read a single line and then finish the lambda invocation with success context.done(...) and with return value whatever the line we received was.

It is very important to understand that this solution is not limited to single line responses. You can have your own protocol that spans multiple lines of response from the application, or even going down to binary data instead of line-by-line.

Finally, once we set our handler callback, we write on standard input of the subprocess the input to our Racket application. Here I just pass a JSON serialised object containing both the event and the context of the lambda invocation, which should cover most of your use-cases. If you want to avoid JSON deserialisation in your Racket code, then you can parse the event in this wrapper code and just pass a line of comma-separated values down to the application. Again, here I use a single line of input, but you can easily adapt this to span multiple lines of input or binary data.

There is also an additional callback that you can set, the handlerProcCloseCallback(). This callback is called every time the subprocess’s standard input and output streams are closed (usually when the subprocess terminates). For example, you can use this callback to re-spawn the closed subprocess and speedup future invocations, as shown below.

proc.handlerProcCloseCallback = (code) => {
    ensureProcRuns({resetCallbacks: true});
};

Compile and bundle your code

  • Copy the Racket code from above and save it in a file named application.rkt

  • Compile the Racket code into a standalone binary

    raco exe --orig-exe application.rkt

  • Copy the Node.js code from above and save it in a file named wrapper.js

  • Bundle everything together zip bundle.zip application wrapper.js

You can upload bundle.zip now to your lambda function and test it.

Make sure that your lambda function has the following configuration:

Configuration Property Value
Runtime Node.js 6.10
Handler wrapper.handler

Test around with different memory configuration to find a good balance between latency and cost for your application.

Conclusion

As you can see, we did not invent quantum computing. We just applied old-fashioned programming principles to AWS Lambda. We have our custom application logic written in Racket, or any other compiled language, that accepts data from standard input, and writes to standard output. Anybody used Linux command line tools before? Yes, it is the same!

This solution is very flexible and allows you to either go with a minimalistic protocol of single line input and single line output as I have done above, or even with a super complicated custom protocol. After all, it’s just standard input and standard output.

The most fascinating thing about this approach is that now your custom code can run in AWS Lambda with super low latencies!

SEO tags: #AWS #Racketlang - #AWSLambda #Meets #Racket

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Self-hosted git repository for privacy and control]]> https://www.lambrospetrou.com/articles/self-hosted-private-git/ self-hosted-private-git Wed, 13 Sep 2017 00:00:00 GMT Problem

Few months ago I started receiving the following warning from Github, Your GitHub academic discount coupon has expired, and if I wanted to continue working with my private repositories I would have to pay. In addition to this, for the last couple of months I was investigating and trying different products for personal and private note taking. I found a lot of interesting products that were integrating with git services like Github, or Gitlab and I was really close to settle with them, but I went deeper into their paid offerings and they actually did not provide encryption at rest, so it was a no-go.

Of course, someone can say they are used by millions, so why not trust them. Well, I don’t, and in conjunction with their availabilities issues I decided to find alternatives. Actually, while I was writing this article Gitlab was out of service with the website returning 500 internal server errors.

My next solution was a custom website I would develop that would have a super-simple UI, including an editor and a list of all my notes, which would use either Google Drive or Amazon S3 as it’s backend, and since I would code the whole thing I would add client encryption to my super important notes (like passwords).

I started looking around for websites similar to this, and found plenty, but none provided the privacy I wanted since they would pass my data through their servers, and although they would say they don’t look at them, well I was skeptical! I found some very good products though which I would gladly use if it wasn’t for the privacy issues, like Write-box, Standard Notes and Dynalist.

So long story short, nothing suited my needs which are summarised below:

  • Privacy of my data with encryption at rest
  • Accessibility and Availability of my notes from my laptops and my smartphone

I was close to start implementing my solution, but then I looked closely into every coder’s friend, Git.

Solution

I decided to use git repositories for storing my notes, and files. After investigating several self-hosted solutions, including gitlab’s, and bitbucket’s server offerings, I settled on an open source super lightweight self-hosted git server, Gitea. Gitea is written in Go which allows for amazingly low resource consumption (it uses less than 0.3% CPU on my T2.nano EC2 instance).

In addition to the self-hosted git server, I found out about the git bare repos which basically act like a git server when hosted on a system accessible over SSH. I use these bare repos for super simple stuff where I don’t need the accessibility over my phone, like coding side projects.

Git bare repositories

Due to the design of git, you can have a directory acting as the remote location of all your working directories, where you can push and pull your files.

How to setup a git repository which can be used as remote location?

  1. On the server (EC2 instance, or any other system accessible through SSH) run the following command to create the git directory:

    mkdir -p $HOME/git-repos/<REPO_NAME>.git && cd $HOME/git-repos/<REPO_NAME>.git
git init --bare

    The first command creates an empty directory where our files will be and the second one initialises that directory to hold a git repository. The --bare argument is the important bit, which tells git to create a repository without a working directory, meaning that nobody will use this directory as his workspace to work on the files. It will be simply used as a remote location for other clones of this git repository.

  2. On any system that you want to work with the above repository just clone it and work with it as you would with any other git repo.

    git clone username@serverHostname:~/git-repos/<REPO-NAME>.git

The good thing of git bare repositories is that they can be used from any system that has access to the server hosting them. I know that lots of people are using this approach but instead of hosting their bare repos on a server, they store them in Dropbox or Google Drive. They achieve the accessibility and availability aspect as long as they keep their systems in-sync, but the privacy aspect still resides in the amount of trust you have on Dropbox and Google. BTW, I am a huge fan of Google services like Google Drive and Google Keep, which I use for pretty much all my storage and note taking needs. However, there are some stuff for which I need some extra privacy (i.e. nuclear codes)!

Self hosted git server

As I said, my git server of choice, at least for the time being, is Gitea, especially due to the low resource consumption on my server. It provides a nice Github-like web interface which you can use to read and edit your files using any web browser, and also has simple organisation of the repositories it creates just like any normal git bare repo which you can use like explained above if you want to bypass the server.

It is amazingly simple to setup as described in https://docs.gitea.io/en-us/install-from-binary/.

# Download the git server
wget -O gitea https://dl.gitea.io/gitea/1.0.1/gitea-1.0.1-linux-amd64 && chmod u+x gitea

# Run the server
./gitea web
# Or Run the server in the background
nohup ./gitea web &

Once you have the server running, access it through a web-browser and you will be prompted for the first setup, which will ask you to register a user (automatically becomes the admin user as well), and to choose your database (SQLite3 should be sufficient without other dependencies).

Obviously, you need to have access to your server over HTTP or HTTPS preferably.

As a side note, I have to mention that Gitea stores all the git repositories you create through its web interface in a directory you define during the first setup and these repositories are just like any git bare repository you create on your own. In a nutshell, Gitea is just a web proxy to git bare repos that does user management and provides a web interface to them.

Backups and encryption

OK, I achieved availability and accessibility since I can access my git server from anywhere using a web browser or over an SSH connection. We achieved incredibly easy management of my files through any editor and any system that can manage git repositories, without forcing us to use a single website UI. But what happens with encryption and privacy?

In my case, I use a T2.nano EC2 instance as my server which I use for personal projects few years now. The EBS volume backing the instance is using Amazon EBS Encryption so we achieve our encryption needs. The security of the server itself is another topic but following AWS best-practices around IAM security rules it should be fine.

In addition, just to avoid any problems when that horribly dark day comes when my host will die or stop working, I take backups of all my repos and store them in S3 with server-side encryption after every push I do on the repos. I also take weekly backups of the whole EBS which will make it very easy to just spin up another server with the exact configuration if this one fails.

The script I use for my repository backups is as shown below, and it assumes that all the git repositories I create, on my own or through Gitea, are under the $HOME/git-repos path.

#!/bin/bash

set -e

GIT_REPO_DIR="git-repos"
S3_BUCKET="s3://your-s3-bucket-name/git-repos/"

TIMENOW="$(date --rfc-3339=seconds | tr ' ' '_' | tr ':' '-' | tr '+' '.')"
FILENAME="git-repos.$TIMENOW.tgz"

DEST_TAR="/tmp/$FILENAME"

tar -cvzf "$DEST_TAR" -C "$HOME" "$GIT_REPO_DIR"

aws s3 cp "$DEST_TAR" "$S3_BUCKET$FILENAME" --storage-class STANDARD_IA --sse

If you don’t want to get into the trouble of enabling webhooks in Gitea to backup after every push, you can just do periodic backups using cronjobs. Assuming that the above backup script file is located in /home/user/bin/backup-git-repos-s3.sh a simple cronjob to take a backup every day at 20:00 is shown below.

crontab -e

0 20 * * * /home/user/bin/backup-git-repos-s3.sh > /tmp/crontab.backup-git-repos.log 2>&1

I personally use Amazon S3 for my backups cause I already use it for hosting all my static websites and for general backup storage since it’s dirty cheap. You could easily modify the script above to upload the file in Google Drive, or any other preferred service.

Conclusion

The first initial setup for the Git self hosted server and the backup automation might seem more work than just signing up on an online service, but for me every it totally paid off, especially since I already had a server running. I have a super flexible way to keep track of personal projects I don’t want to have public, or to store sensitive information without being afraid of prying employee eyes. And with the backups, in case something goes wrong with my server I have perfectly good archives to work with.

Although I solved my problem, I still wish there was a completely client-side web application integrated with Google Drive or Amazon S3, which will provide capability to encrypt my notes before storing them, without transmission of any data to third-parties (apart of course from the storage service itself).

To be honest, I was super amazed that there are so many online products integrating with Github, Dropbox, Google Drive, and others, but still none of them provides privacy and client-side encryption before sending the data over. Therefore, I still plan to implement that super-duper web application, so keep an eye for it over the next coming months :)

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Solve the eight queens problem with Elixir]]> https://www.lambrospetrou.com/articles/elixir-queens-recursion-comprehension/ elixir-queens-recursion-comprehension Mon, 01 May 2017 00:00:00 GMT Problem

While I was learning Elixir I wanted something to use the for-comprehension to solve a problem. I remembered the Eight queens puzzle back from my college courses so I decided to give it a go.

I adapted the problem a bit, so my solution takes as inputs the following:

Inputs
======
n: number of queens to position on the board
m: size of the board side

Outputs
=======
List of all the solutions (List[List[int]])

Naive Solution

The solution is simple, and uses backtracking to iterate over all possibilities and select the valid ones.

defmodule Queens do

  @doc """
  Given n number of queens and m the size of the checkerboard, find all solutions to 
  position each queen so that it does not collide with any other queen vertically, 
  horizontally or diagonally.
  """
  def solve(0, _m), do: [[]]
  def solve(n, m) do
    for done_queens <- solve(n-1, m),
        avail_pos <- (Enum.to_list(1..m) -- done_queens),
        safe_pos(avail_pos, done_queens, 1), 
      do: [avail_pos | done_queens]
  end

  defp safe_pos(_, [], _), do: true
  defp safe_pos(pos, [queen | queens], distance) do
    (pos != queen + distance) and 
    (pos != queen - distance) and 
    safe_pos(pos, queens, distance+1)
  end

end

Obviously, this is not the fastest algorithm to solve this problem (exponential complexity), but it shows how elegant the solution can be using Elixir’s comprehensions.

Sample output in iex:

iex(47)> c("queens.ex")
[Queens]
iex(48)> :io.write Queens.solve(4, 4)
[[3,1,4,2],[2,4,1,3]]:ok
iex(49)> :io.write Queens.solve(3, 4)
[[2,4,1],[1,4,2],[4,1,3],[3,1,4]]:ok
iex(50)> :io.write Queens.solve(5, 5)
[[4,2,5,3,1],[3,5,2,4,1],[5,3,1,4,2],[4,1,3,5,2],[5,2,4,1,3],[1,4,2,5,3],[2,5,3,1,4],[1,3,5,2,4],[3,1,4,2,5],[2,4,1,3,5]]:ok
iex(51)> :io.write Queens.solve(1, 5)
[[1],[2],[3],[4],[5]]:ok

Demo: http://elixirplayground.com?gist=688be58a64712a172878a58683ed0eda

Conclusion

Every day I learn more and more Elixir (and Erlang) and I can say that it’s among the few languages that managed to keep me interested and excited for more than 3-4 months (looking at you Scala and Python).

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[AWS S3 sync - only modified files, using git status]]> https://www.lambrospetrou.com/articles/aws-s3-sync-git-status/ aws-s3-sync-git-status Sat, 24 Sep 2016 00:00:00 GMT Problem

As you know I build this website using my custom site generator builder so the whole website is being re-created every time I change something. Similar tools are widely used by many people, like Jekyll.

Then I use the AWS S3 sync command to update the public version of the site in my S3 bucket. The only problem with this is that since the local website is fully re-built every time it seems to be newer than the remote website in the S3 bucket (due to newer modification timestamps), resulting in extra uploads for all files.

Git Solution

The easiest solution to solve this problem is to use git to handle diff changes and then just pass along the modified files to the aws s3 sync command that will sync them against the remote S3 bucket.

So, long story short, assuming you have the site’s directory git-versioned the following script will sync the directory with the remote S3 bucket, including adding new files, removing deleted files, etc.

#!/bin/bash
set -ex

FILES=()
for i in $( git status -s | sed 's/\s*[a-zA-Z?]\+ \(.*\)/\1/' ); do
    FILES+=( "$i" )
done
#echo "${FILES[@]}"

CMDS=()
for i in "${FILES[@]}"; do
    CMDS+=("--include=$i""*")
done
#echo "${CMDS[@]}"

echo "${CMDS[@]}" | xargs aws s3 sync . s3://www.lambrospetrou.com --dryrun --delete --exclude "*"

Important

You have to remove the --dryrun option in order to actually apply the changes remotely, otherwise it will just fake them.

Explanation

The important part of the above script is the --include and --exclude filters. The order of the filters matters, that’s why we have the exclude first, and the includes last. In case the exclude was last nothing would be updated.

The two for-loops generate the required --include=FileX arguments, which are expanded using the "${CMDS[@]}" trick. Then xargs takes care of sending them as last arguments to the aws s3 sync command, also taking care of very long list of files that exceed the command line length limit.

In addition, I have to use git status instead of git diff otherwise new files will not be synced, since they are not part of the index tree.

Conclusion

Using git along with aws CLI it’s very easy to maintain my website and only upload the real diff, modified files, each time. One can imagine that this can be used in a much more advanced scenario with Github webhooks integrated with AWS CodePipeline or any other CI tool that will release your website automatically.

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Transfer your contacts to an old Nokia Symbian S40 mobile phone]]> https://www.lambrospetrou.com/articles/nokia-symbian-s40-contacts-transfer/ nokia-symbian-s40-contacts-transfer Thu, 30 Jun 2016 00:00:00 GMT Every now and then we have to change phones and transfer contacts from one device to another. It’s a trivial thing to do nowadays with Android, iOS and Windows Phone, but try to do it on an old Nokia Symbian S40 phone. It’s 2016 and this simple thing is just fuxxing too complex. And it’s standards inconsistencies and Nokia to blame!

Problem

I have my contacts in Google Contacts, how they got there from an older S40 device is an even more fascinating story but let’s leave it for another post. Now I just want to import these contacts into a Nokia 301, which runs Nokia Symbian S40, and yes some people still use this kind of phones, obviously not me :)

Solution 1 - Nokia PC Suite

Nokia PC Suite was a nice tool by Nokia, if and only if it worked with your phone. In order to make it to recognise the device I had to install specific versions of the software (Nokia PC Suite version 7.1.180.94) and specific version of the Nokia Connectivity Cable driver (version 7.1.182.0).

OK, we are connected and we can manage the device through the PC suite. Now, what?

First of all, I needed to have the contacts in a form that can be imported somehow by the PC suite, so I exported all the contacts from Google to .csv and .vcf formats just in case one did not work.

There is a handy option in PC suite when you open the contacts section:

File => Import

which allows you to select any .csv, .vcf file to directly import into your phone. Of course this would be too easy :) so neither the .csv nor the .vcf import worked using the files from Google Export.

The problem with .csv is that Google’s csv format follows a different structure compared to Nokia’s PC suite so that’s a no-go. The problem with .vcf is that this one file was containing ~1000 contacts but the import functionality was only importing 1 contact (the same happens when you send this .vcf file directly to the phone via Bluetooth).

After spending several hours going through myriads of forums with people asking for a solution I found an article written in 2009 that provides a solution to split this single .vcf file into multiple .vcf files, one for each contact. The tool mentioned in the article, which I also used is vCard Split by Philip Storry.

Very important

  • In step 3 (see screenshot below) you have to select Force version 2.1 otherwise Nokia PC suite will complain and fail to import the contacts. And also select/check the Remove Type information option because Google puts a TYPE section in each .vcf which is again not recognizable by PC suite.

vCard Split screenshot

Awesome, we have all our contacts splitted in a single .vcf file per contact. Now the last thing to do is just select all these .vcf files, COPY them (CTRL + C) and then PASTE them (CTRL + V) into the PC suite contacts section, or simply use the File => Import wizard and select all the .vcf files from the opened window.

Great, the tool seems to be importing the files and finishes successfully. But, there is a catch :) It seems that Nokia only cares about english/latin characters and any Greek character is converted into a weird symbol which makes your contacts useless.

I searched for quite some time to find a way to bypass this limitation but it’s not possible, so PC suite is out of the question. If you want Greek contact names, Nokia PC suite is a BIG FAIL.

Solution 2 - Nokia Transfer app and an Android phone

To be honest this was the first solution I tried but it instantly failed so I went on with PC suite, until that one also failed and I returned to this one. This time I noticed that it was just a small step missing to make it work.

Nokia bundles with the Nokia 301 an application named Tansfer which you can also manually install on your own by downloading the .jar/.jad setup files and using PC suite to install them on the device. I manually installed the latest version Transfer 1.0.11.

This application’s functionality is transferring data from some phones over to your device via bluetooth. I have a Motorolla Moto X and Moto G3 so I tried to use this and transfer the contacts directly from the Android phone to the Nokia phone. The problem was that I was always getting the error This service is not supported by the phone with both Androids, which was weird.

Then I went with PC suite route but we know how that ended up. So I decided to give another go on this simple method. This time, the magic was that I deleted the bluetooth pairing between the phones on both of them.

The following steps did the job:

  1. Turn on bluetooth on Moto G3
  2. Turn on bluetooth on Nokia 301
  3. On Nokia 301 go to bluetooth trusted devices and search for new one. Select the Moto G3 and accept the prompts in both phones, checking/selecting the option shown in Moto G3 which says Allow Nokia 301 to read contact information.
  4. Open Nokia Transfer app on Nokia 301 (inside the Applications menu option)
  5. Select Add new device and wait until Moto G3 appears and select it
  6. Select Moto G3 from the populated list
  7. Select Import contacts
  8. Wait for the import to finish :)

And now I finally have all the contacts from Google into the Nokia 301.

Conclusion

If you have a compatible device with Nokia Transfer app it is by far the easiest way to import your contacts into a new Nokia S40 phone, but remember to delete the pairing and then do it properly allowing access to contacts.

If you want to use PC Suite, unfortunately you cannot import contacts with Greek names, and most probably any contact name with non-ANSI characters.

But please, upgrade your phones to something modern, preferably Android :)

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Update Route53 record set with EC2 instance public IP for a DIY load balancer]]> https://www.lambrospetrou.com/articles/aws-update-route53-recordset-diy-load-balancer/ aws-update-route53-recordset-diy-load-balancer Wed, 29 Jun 2016 00:00:00 GMT In this small tutorial I provide a code snippet that allows you to update a record set in Route53 with the public IP of an EC2 instance programmatically.

Why? Just use an Elastic Load Balancer!

Elastic Load Balancing is an awesome service that handles load balancing of the traffic to your instances very well. But, it comes with a price, literally. An ELB charges a small amount for every hour running, pretty much like a T2.small instance (as of time of writing). This is dirty-cheap when you have tens or thousands of instances behind it, but for 1-2 T2.nano instances it might seem overkill.

I play around a lot with AWS and I create small projects, websites mostly, where a single node is more than enough to handle all the load, thus paying for an additional load balancer is too much.

Moreover, there are many people that have use cases where ELB is not the right solution and they prefer to have Route53 act as their load balancer. Loggly team wrote a very nice article describing this approach and I recommend it for anyone interested in real-world advantages.

Alternatives?

There are several custom solutions to avoid using an ELB. Most of them take advantage of Elastic IPs and Route53 API to programmatically update record sets.

Solution 1 - Elastic IP only

This solution is as simple as possible and only applies to single instance applications. When the EC2 instance boots you provide a user data script which will associate your Elastic IP (you should create this before booting the instance) with the instance.

Find more information for the exact CLI command at the official EC2 CLI reference.

I recently discovered that this is how single-node configuration of Elastic Beanstalk is implemented (very similar).

Solution 2 - Route 53 only

This solution requires a bit more work from you in terms of scripting but it is more flexible than having an Elastic IP for each instance. Inside the user data script of your launch-configuration you have full access to the AWS CLI. Therefore, you can pretty much do anything!

Let’s assume that you have an application at test.lambrospetrou.com served by a single instance. You want to programmatically update the A record of this record set to point to any new instance being created by your max-1-min-1-desired-1 auto-scaling group.

It turns out it is very simple to do :)

First of all we need to find the public IP of the running instance. AWS provides several metadata information to every EC2 instance and public IP is one of them.

curl http://169.254.169.254/latest/meta-data/public-ipv4

The next step is to update the test.lambrospetrou.com record set. It is pretty easy to navigate through the Route53 - ‘change-resource-record-sets’ documentation which we find that the JSON we need is pretty much the following:

{
  "Comment": "Update the A record set",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "test.lambrospetrou.com",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "127.0.0.1"
          }
        ]
      }
    }
  ]
}

In the above JSON you can see that we want to do an UPSERT (update) for the test.lambrospetrou.com record set, with type A since we want to point to an IP address, and with value 127.0.0.1.

In order to do the record set update you need the following command (assume that the above JSON is the content of a file named update-route53-A.json):

aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --change-batch file://./update-route53-A.json

In the above command you have to put the correct Hosted zone id where the record set resides. You can find this in the Route53 console or with the following command:

aws route53 list-hosted-zones-by-name

If you try and play with the above command you will notice that the update is pretty much instant, which means that as soon as the instance is up and running, your application will be available using the domain.

There is one variation of the above command which accepts a string instead of a file, which makes it easier to use through scripting. The only difference is that the JSON we examined above needs to be enclosed in a field named ChangeBatch:

{ "ChangeBatch": { /*INPUT_JSON_LIKE_BEFORE*/ } }

and the command to use this JSON string is as follows:

aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --cli-input-json "$INPUT_JSON_STR"

To summarise, these are the only two commands we need to do what we want:

  1. Get the public IP of the running instance

    curl http://169.254.169.254/latest/meta-data/public-ipv4

  2. Update the Route53 record set

    aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --cli-input-json "$INPUT_JSON_STR"

Of course you will want to do some more scripting to replace the 127.0.0.1 value with the proper IP and programmatically find the hosted zone id of your domain. The following snippet is a quick/hacky way of accomplishing this. Feel free to use it if you are bored to come up with a better one :)

#!/bin/sh

if [ -z "$1" ]; then 
    echo "IP not given...trying EC2 metadata...";
    IP=$( curl http://169.254.169.254/latest/meta-data/public-ipv4 )  
else 
    IP="$1" 
fi 
echo "IP to update: $IP"

HOSTED_ZONE_ID=$( aws route53 list-hosted-zones-by-name | grep -B 1 -e "lambrospetrou.com" | sed 's/.*hostedzone\/\([A-Za-z0-9]*\)\".*/\1/' | head -n 1 )
echo "Hosted zone being modified: $HOSTED_ZONE_ID"

INPUT_JSON=$( cat ./update-route53-A.json | sed "s/127\.0\.0\.1/$IP/" )

# http://docs.aws.amazon.com/cli/latest/reference/route53/change-resource-record-sets.html
# We want to use the string variable command so put the file contents (batch-changes file) in the following JSON
INPUT_JSON="{ \"ChangeBatch\": $INPUT_JSON }"

aws route53 change-resource-record-sets --hosted-zone-id "$HOSTED_ZONE_ID" --cli-input-json "$INPUT_JSON"

As an additional note, the above example updates the A record with a single IP. You can easily adapt the script to retrieve the current IPs of the record set and append the new one to them. This way you can even achieve simple round robin load balancing between nodes exploiting Route53’s weighted routing. Imagination is the limit to what you can achieve :)

Solution 3 - Route53 & Lambda

Another interesting and even more flexible way of achieving what I explained above, and way much more, is utilising AWS Lambda functions. The combination of Lambda with events and Route53 is super-powerful and can implement very complex configuration updates that are usually very difficult.

A lot of people are already playing with Lambda for doing network configuration changes triggered by autoscaling. For example, instead of doing the update of the Route53 record set in the user data section of the EC2 launch-configuration, you could setup a Lambda function to be invoked when a new instance has been created (or terminated) and do your changes using the official AWS SDKs available in Lambda functions.

Many AWS tutorials are available in the AWS blogs but you can find some in the References section below.

Conclusion

The above solution is by no means comprehensive or suitable for everyone and every use-case. Most of the time you will want to use Elastic Load Balancing which provides cross-AZ balancing, monitoring, and transparent load balancing in front of several auto-scaling groups.

In some cases though you want to keep it simple or cheap, and this is where these solutions are preferred!

Feel free to contact me for any mistakes you find in the snippets above or if you have a better solution for the aforementioned problem.

Happy AWS Clouding :)

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Banana or Human, and Marginal Degradation]]> https://www.lambrospetrou.com/articles/banana-or-human-marginal-degradation/ banana-or-human-marginal-degradation Sat, 11 Jun 2016 00:00:00 GMT A few months ago I attended a talk given by Monotype’s Steve Matteson who designed awesome fonts like Open Sans, Android’s Droid family and Microsoft’s Segoe family among others, and by Mark Boulton who is now Design Director at Monotype and founder of Mark Boulton Design, which was acquired by Monotype :)

The talk was pretty great and as you can imagine from their background, it was about Typefaces and their importance in a brand’s or a product’s value. Yes, I am a software engineer but I do care about the design and the aesthetics of my work! As much as I hate Apple, Steve Jobs was the leading exemplar of the fusion between technology and design, and will always be one of my role models.

Anyway, the one thing that stood out and impressed me the most out of the talk was Mark’s speech about Marginal Degradation. I found it so basic and simple but at the same time so powerful that it just sticked to my mind. After a few days I decided to read more about it and found a related article in Mark’s blog which is basically a scripted version of what he said. I always paid attention to the smallest detail, but after reading and hearing about this I just made it a habit! I will give you a brief summary below but I encourage you to read the whole article.

Banana vs Human DNA

Here’s a few fun facts… - Gorillas share 98.4% of our DNA - Goldfish share 68% - Bananas share 50%

Bananas. Are 50% the same as us.

Read the above quote, then read it again, and then again one last time!

Only 1.6% of DNA differentiates us from gorillas and we are half-way to becoming bananas! Don’t get all fancy science genius with me now, I don’t know how much really that is in DNA world, but the fact still holds. Even the slightest marginal degradation of our DNA can make us inferior.

Now consider this to whatever you do. I write code as a living, you might be writing literature, or you might be designing the next website for the most awesome startup ever. You have to pay attention to detail, even the smallest one! Do not let anyone tell you “Just do it quickly for now, the manager wants it urgently. We can revisit this later.” No, you will not revisit it later and you will not fix this little sucker. It is there and it will stay there, degrading and sucking out the value and quality of your product.

Your brand or design is supposed to be a human, but people perceive it as a gorilla. Or a banana.

You have to really think the consequences of everything you sacrifice by not giving 100% of yourself in doing something. It might seem like a tiny thing at the moment, but if you take into account all those little tiny things you sacrificed over the years, you will find out how much you deviated from your original goal.

Nobody wants his product or brand to be seen as a banana, when it was supposed to be a human!

Takeouts

  • Pay attention to the details!
  • Do not sacrifice quality for a moment’s weakness or lack of time!
  • Always have in mind the original goal/target of what you want to achieve, otherwise every little deviation might lead to something entirely different!

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Keyboard shortcut for opening context menu]]> https://www.lambrospetrou.com/articles/keyboard-shortcut-for-context-menu/ keyboard-shortcut-for-context-menu Tue, 07 Jun 2016 00:00:00 GMT This is going to probably be the shortest article I wrote, or will write!

Have you ever wanted to open the right click menu on something without lifting your hands from the keyboard? I did a lot of times, especially when navigating through videos wanting to open them with the non-default application (which can be done with the Open with option in the right click context menu).

Well, you can do it with the following keyboard shortcut:

Shift + F10

Warning: It does not work in all the applications, but it does in the file manager I use which is enough for me :)

Alternatively, you can set keyboard shortcuts to trigger mouse events including right click, but this is out of this article’s scope.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Backup your Wordpress and serve it as a static website]]> https://www.lambrospetrou.com/articles/wordpress-backup-static-hosting/ wordpress-backup-static-hosting Sun, 05 Jun 2016 00:00:00 GMT A few months ago I started migrating all my projects and websites into AWS. The last thing standing was my old wordpress blog (hosted at mastergenius.net, not alive anymore). Today, I decided to make a final backup and create a static version out of it and ditch the VPS I have running just for that.

It turned out to be extremely easy :)

Backup all content

Apart from serving the content as static website I wanted to have the posts in a form that can be easily imported later back in Wordpress and also a format which is easy to parse and read using a text editor.

I just wanted the text, which is the heart of any article anyway, so I just did the usual export feature of Wordpress as described in the official export documentation.

A full folder copy is recommended too, you know just in case I want to revisit any files, images or code!

Serve as static website

There are a lot of ways to dump an active wordpress installation and several plugins that allow you to do this conversion easily. I just took the simplest way and used the well-known wget Linux command line tool (original Quora question link with the command is in the References section).

wget -k -K  -E -r -l 10 -p -N -F --restrict-file-names=windows -nH http://active-wordpress.domain.com

After executing the above command I deleted all the unnecessary files, stripping down the whole site in some MBs.

I just uploaded the static files to Amazon Simple Storage Service - S3 and that’s all!

Zero-cost hosting :)

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Fun with Typescript, Webpack and ReactJS]]> https://www.lambrospetrou.com/articles/webdev-typescript-webpack-react-numerology/ webdev-typescript-webpack-react-numerology Sun, 22 May 2016 00:00:00 GMT I decided to play around with Typescript instead of BabelJS, which I already tried in the past, for this weekend’s project.

Reasons for Typescript

During the last year I explored several Javascript languages/flavors (transpilers, etc.). If you read my previous articles you should know that I also like Dart. I started using it in 2014 along with AngularDart throughout its journey to the stable version 1.0, always going through refactorings and fixing breaking changes. But Dart made me feel again productive and gave me a sensible platform to do front-end development. Unfortunately, it did not pick up as many fans as Google (and me) wanted so it was kinda left behind.

I really believe that the future lies in languages that compile to Javascript or even something else, WebAssembly maybe? (WebAssembly and Article )

I already gave a go to Babel, so this time it was time for Typescript. Why choose Typescript and not stay with Babel is justified by the points below.

  • I love static typed languages :) and Typescript is much closer to that rather than JS
  • I prefer compilation errors much more than runtime exceptions
  • I believe it will be alive for at least 3 more years since it is backed by Microsoft and Google and is used as the main language for Angular 2
  • Better IDE support (if I decide to use one)

Project

I wanted to not spend more than 1-2 days so I have chosen something super simple to implement. I decided to develop a numerology calculator of your life path number :) Fancy stuff!

Calculate your life path number at the demo page

Build Process

My main goal with the project was to come up with a build process using Typescript with ES6, Webpack and ReactJS. I also wanted to use Autoprefixer and SASS so I added Gulp to the mix too, but I tried to keep it at a bare minimum.

The final process is explained in the following sections.

Code

I wanted from the beginning to consolidate the code building into one tool, and the decision was pretty easy since Webpack is the big winner (at least for now).

The requirements are simple:

  • Write my code in Typescript
  • Use ES6 features
  • Support browsers that only have ES5 support
  • As a bonus I also wrote the gulpfile in Typescript and used a hack-around to use it to actually compile the Typescript code :)

For ReactJS I needed to install react and react-dom along with their typings to allow Typescript to resolve the React types. Pretty smooth process and I also found out something I did not know before. With webpack I can use the npm modules during development but do not bundle them along with my app code to keep the footprint small. By specifying that these modules are external I just need to use script tags in my HTML to import the public ReactJS libraries and take advantage of CDNs and local cached versions.

npm install typescript typings react-dom react --save-dev
typings install dt~react dt~react-dom --global --save

In order to be able to use some of the ES6 features, like Object.assign(), I had to include es5-shim and es6-shim in my HTML code right before I imported the Typescript compiled bundled which targets ES5. I also needed to install the typings for ES6 in order to stop Typescript compilation errors when using ES6 features in the code.

typings install dt~es6-shim --global --save

So, the Typescript compiler will use the typings and compile the code into ES5-valid Javascript and then Webpack will take all the files and output 2 bundles. I like to have separation between the app code which is Typescript and other JS code I might have, so I compile a bundle for the Typescript code ts.bundle.js and one for normal JS-code js.bundle.js.

Use Typescript for the Gulpfile

I could have used regular ES5 Javascript to write the Gulpfile but I wanted to play more with Typescript. I spend some time to be honest searching for solutions for this and I ended up following this awesome guide adapting it where needed.

SASS and Build

No surprises here! Gulp is used to clean the build directory, to use Autoprefixer to prefix new/useful CSS with browser-prefixes and to build the final CSS using SASS.

Conclusion

I spend some time back and forth with the Typescript documentation, expected since this is the first time I tried it, but I liked it. I am going to stick with this for a while until I find something much more productive and powerful, ScalaJS might be that one :)

Finally, I liked ReactJS too, and this is the second fun-project I do using it, but since I am an Angular die-hard I have to try the stable version of Angular 2 (which just released RC) to decide if there is a winner between them.

If you have a better pipeline for the above tools please contact me :)

Source code is hosted at Github

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[AWS Certified Developer & Solution Architect Associate Certification Tips]]> https://www.lambrospetrou.com/articles/aws-certification-associate-dev-tips/ aws-certification-associate-dev-tips Sun, 15 May 2016 00:00:00 GMT I wanted to write this article for a while now but I waited until I actually passed the certification exam before I started giving tips to others :)

This article is mostly a reference for myself, but it might also be helpful to anyone preparing for the AWS Certifications. I used the following resource material myself over the last couple of months and since I passed the exam with a pretty high mark I can assume they are good enough to recommend them :)

First of all, you have to bookmark the official AWS Certification Preparation Guide.

AWS videos

The first thing I started doing when preparing for the exams, Solutions Architect and Developer, is watching all the re:Invent videos from 2015.

It might seem that they are just too many to watch or that you will spend huge amount of time, but trust me, they are going to help you more than any documentation or tips or online trainings. They will not only give you insights for individual services but also architectural design best-practices which you can apply into your own work environment even if you are not AWS-hosted.

Personally, I watched pretty much all the 2015 re:Invent videos over the span of 3 months, but you can just focus on the fundamentals and the deep dives and skip the war stories from several companies.

Below you will find the links to the AWS re:Invent 2015 playlist on Youtube and to the AWS Summit 2016 - Chicago playlist which is just a few weeks old :)

The re:Invent videos are marked with a department and a 3-digit code denoting the pre-requisite knowledge you need to have in order to being able to follow the speaker. Remember the college course naming scheme? It’s the same!

AWS re:Invent 2015

You can find the videos at AWS re:Invent 2015 Youtube playlist

Some of my favourites are (no particular order):

The above are just a few selections to start with and by no means conclusive. I also recommend you watching re:Invent 2014 and 2013 videos especially deep-dives.

AWS Summit 2016 - Chicago

You can find the videos at AWS Summit 2016 - Chicago

This videos are not as structured as the re:Invent ones but you can as easily filter out the service-oriented ones from the war-story-oriented ones.

Again a few selections:

There is some overlap with re:Invent videos in most of the summit ones but there is also additional valuable content.

AWS Whitepapers

Whitepapers are my second favourite resources after the re:Invent videos. They present some of the most important topics of deploying services on AWS and provide best-practices and pitfalls to avoid in the process.

You can find them all at AWS Whitepapers.

Recommended reading:

Documentation and FAQs

Another great resource is obviously the documentation of each service and its FAQ. I would strongly advise you to go over the FAQ of the most important services, depending on your exam, and if you have time then go over the docs.

I have to warn you :) FAQs are a must!!!

You can find links to most of the FAQs at https://aws.amazon.com/faqs/ Or you can visit the FAQ of the service you want directly by visiting the appropriate link, aws.amazon.com/SERVICENAME/faqs/. For example, the Simple Storage Service (S3) FAQ will be at aws.amazon.com/s3/faqs/.

Personally, I read the FAQs while travelling in the aeroplane or in the tube/buses. As a result, I created a small script to extract the main content out of the website of each FAQ and I put them on my Kindle :) resulting in a much better and easier read!

You can find the small script on Github: AWS FAQ client

AWS Training classes

The official AWS Certification Preparation Guide suggests to attend training classes depending on the exam you want to pass.

I attended two of them so far, the Developing on AWS and the System Operations on AWS. I have to admit that the developing one apart from some advices and tips from the trainer himself it was very basic and I would not recommend it to someone that already uses AWS. However, if you are just starting with AWS it is a great introduction to the core services. The SysOps training on the other hand was amazing. I highly recommend this training to anyone because it covers things from advanced monitoring, to tagging, to cost-optimization, to custom AMIs and deployment options and pretty much things that I doubt you will ever do on your own or if you are not already working as a systems administrator.

Update @2016-05-30

Last week, I attended the Architecting on AWS training course. I would really recommend it to anyone that will attempt the certification exam since it is a very nice overview of all the architectural aspects of AWS. It goes over specific use-cases, optimizes and finds bottlenecks of the architecture for certain scenarios which requires you to know about several services and how they can be used together.

Overall, if money is no problem (most companies cover the expenses anyway), I would really recommend you to attend these 3-day training courses. But, it really depends on your knowledge and expertise level.

AWS Qwiklabs

I highly recommend you to do as many AWS Qwiklabs as you can. They range from quick to long practical labs on most of the services and basic concepts you will need for the exams and for actual work on AWS.

Start by doing the free ones first, but I really encourage you to try some of the advanced ones too. Even better if you could convince your company to pay for them. Not only they cover in-depth concepts but they allow you to use expensive services that would be cost-prohibitive to use and play with using your own personal account.

Projects, projects, projects

Most Important

As with anything you want to learn, actual practice provides the best learning experience.

I did several small projects, and wrote about them in previous articles, where I actually used most of the services covered by the exams. This provided me with invaluable knowledge and insights as to how some services work which I could not acquire just by reading stuff. After all, the exam is called Developer Associate because you need to be able to develop and build systems, not just know about them :) I can assure you that the exam contains questions that you cannot answer correctly just by reading the docs.

Qwiklabs can significantly contribute to this if you opt-in for the advanced labs but again, from personal experience, if you do not build something from scratch on your own you do not really learn.

Conclusion

I provided the most important resources I used to prepare for the exam and get myself acquainted with most of the AWS services. I am pretty sure if you follow my advices you will have great results, but by no means consider this an exhaustive list of content or tips.

Watch as many videos as you can from all the re:Invents, do as many qwiklabs as you can, and most importantly try to do small (or big) projects utilising a variety of AWS services.

The content of the different examinations has a significant amount of overlap, but focuses on different aspects of each topic. For example, the developer exam will concentrate on performance, API, and implementation details of a service whereas the architect exam will focus on orchestrating and utilising the correct services to solve a problem.

Good luck :)

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Yet another Spito re-write, on the Cloud (AWS)]]> https://www.lambrospetrou.com/articles/yet-another-spito-rewrite-aws/ yet-another-spito-rewrite-aws Mon, 25 Apr 2016 00:00:00 GMT Finally, I found some time to re-write my super-duper URL shortener yet once again :)

As part of my preparation for the AWS Certifications I plan on taking, I wanted to make an architectural re-design of Spi.to completely cloud-based on AWS. Last weekend, I finally convinced myself to stop watching shows on Amazon Video and did it.

Architecture overview

In the following diagram we can see an overview of the application’s architecture. It is is one of the simplest applications you can make in a weekend but at the same time it allows you to use many cloud services following best-practices in order to achieve fault-tolerance, high availability and durability, which is similar to what you would do in a real super-scalable service.

Spito Architecture Overview on AWS

You can see that it is a pretty basic setup but without sacrificing performance, availability or durability!

Services

The services I use in this application can be identified from the diagram above but read below for a small description as to the why use each service.

For more information about Route 53, Cloudfront, and S3 regarding hosting a static website on AWS you can read my previous article.

Amazon Route 53

Route 53 is a cloud-based DNS management service which I use to manage my domain spi.to. I do not use advanced features of the service for this application but you should check it out because it is simply awesome.

Amazon Cloudfront

Cloudfront is one of those services that once you understand how it works and play a little with it, you just love it. It is Amazon’s CDN solution but at the same time can act as a reverse proxy to your backend or as a faster gateway to your services instead of going throughout the public network.

I use it in front of my static website server (explained in S3 section) and my REST API service (explained in Elastic Beanstalk section) which are the two origins servicing my application.

In cloudfront I specify certain behaviors for caching depending on the files but I also include some path patterns to direct each request to the appropriate backend (S3 or API). You can see below a snapshot of the rules I have at the moment.

Cloudfront behavior rules

The above rules are evaluated top-down until the path matches one rule and that rule is applied without going further down the rule-chain.

You can observe that the first rule ensures that all /api/ calls are going to our API backend whilst whatever request comes with . (dots) or / (slashes) will go to the website client backend. The fourth rule ensures that whatever request comes with at least one character, I want to match the hash ids now, will go to the API backend. The last rule ensures that the root path spi.to/ without any path (since we covered all other cases) will go to the website client, which is the homepage.

There might be a simpler solution, but I could not find a way to use classes of characters in path patterns. If you find one or I missed something in the documentation please contact me :)

Also, another important thing with Cloudfront that a lot of people ignore is that you CAN use it with dynamic services. In my case the API is strictly dynamic since the spits have expiration dates and they cannot be just cached. You can specify that a path will have no caching, which means that you will just use Cloudfront as a proxy to your service, and as I said before it might be beneficial to your users because the communication to your servers will be done inside the AWS network which has all sorts of optimizations.

Amazon S3 - Simple Storage Service

I use Amazon S3 in order to serve the web client (website) which you can access by visiting http://Spi.to. The client is a static web application written in Dart which consumes the Spito API (REST API explained below).

Amazon S3 as discussed in my previous article, Hosting a static website at AWS, is just amazing for static content (especially when combined with Amazon Cloudfront).

Amazon DynamoDB

Amazon DynamoDB is used to store the text or the URL that you upload to the service. I was considering to use S3 again for that but Dynamo is much faster for small texts. I will update the API in the future and use S3 for large-sized text (now the service only allows you to post text up to 128KB).

DynamoDB has very low latency and allows the service to provide super-fast retrieval of the so-called Spits (my naming for the uploaded text) and also provides durability and high-availability of the data.

AWS Elastic Beanstalk

Elastic Beanstalk is a tool that I just learnt recently, and I loved it instantly. Super-easy to use and you get all the benefits of autoscaling and custom bootstrapping out-of-the-box. You, as a user, just need to upload your source code or binary of the application.

Beanstalk handles the Spito API servers inside a managed auto-scaling group which by its turn is behind a managed elastic load balancer, all handled by beanstalk itself.

For this service I use T2.nano instances in order to keep the costs at a minimal, in case you were interested :)

The Spito API is written in #Go (source code found in Conclusion section).

AWS CloudFormation

AWS Cloud Formation is a future feature I will add to the project in order to automate the infrastructure creation. As developers we love code, so everything needs to be in code :)

Coming soon

Conclusion

Through this project I learnt a ton about Elastic Beanstalk and how to use Cloudfront even for dynamic requests, and also re-used the knowledge and tricks from my last project (the aforementioned article about static websites).

I open-sourced the application, both the website and the REST API, but keep in mind that this is only the application. I still haven’t added the CloudFormation template which creates and bootstraps the required AWS services in an automated way.

Source code

Warning: The client (spitoweb) has been written a long time ago so it clearly does not follow Dart best-practices, however it could be useful to read if you are into Dart too :)

Ah, and before I forget, you can find Spito at http://spi.to.

Any comment or feedback is appreciated.

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Import Gmail messages into Google Keep as notes]]> https://www.lambrospetrou.com/articles/import-gmail-messages-to-google-keep/ import-gmail-messages-to-google-keep Sun, 14 Feb 2016 00:00:00 GMT A few weeks ago a friend asked me for a way to import his Apple Notes into Google Keep. It seems that no easy straighforward way exists as of today so I decided it was a great opportunity to make my first Chrome extension and create my own importer.

It turned out to be a bit more complicated than I expected though, hence no plugin already exists. The reason is that Google as of now does not provide an open API for developers to use in order to import messages which makes the whole project impossible from a programmatible point of view. In the end I managed to overcome this issue and created the extension which you can download from the Chrome store and try it yourself (direct link to Chrome Store).

In this tutorial I will first provide instructions how to use the extension for the impatient ones, and then I will go into more details regarding its implementation and the issues encountered during the process.

How to use the LP Gmail to Keep Importer

The most updated state and instructions about the plugin will always be accessible at its Github repository where you can also download the source code. Any feedback and pull-requests are welcomed.

The following instructions are valid and work with the Google Keep website as of the date of this article.

  1. Install the plugin from the Chrome store: Download LP Gmail Keep Importer

  2. In order for the plugin to work you have to, read again, have to be at the Google Keep website (reason explained below). Visit Google Keep and login with your account.

  3. Once you are at the Keep website, and you have installed the extension you should be able to see the LP G2K icon in Chrome’s toolbar.

  4. Click the extension’s icon in order to open the plugin and you should see a small window like the picture below.

    LP Gmail to Keep Importer - Authorization view

  5. The authorization is required only the first time you use the extension. You have to authorize it to access your emails (read only don’t worry).

    • Click Authorize to be redirected to Google’s authorization page.
    • Click Allow to allow the extension to use your account’s email.
    • The window will stay open with a blank page. This behavior is a bug from Google’s side and they are looking into a fix, but for now just close the window yourself and refresh the page.

  6. Click again the extension’s icon and now you should see the normal view with the text box (from now on this is what you should see since you already authorised it).

  7. In the text box you have to type the Gmail label name you want to import. As you know Gmail organizes your mails, messages, in labels so just find out the label you want to import and type it here or create a new one if you don’t have one. Apple Notes automatically saves your notes as emails under the label name Notes, so use that if you want to import your Apple notes.

  8. Click Import GMail Label. Once the plugin fetches your messages from Gmail, it will show you a small result view with the number of messages found (like the picture below) or it will show an error message if there was a problem during the fetching process.

    LP Gmail to Keep Importer - Results view

  9. Click Start Importing messages in order to start adding your messages as notes into Google Keep. If you have hundreds of messages please be patient and wait until you see a response, either an error or a success message.

  10. When your messages have been imported you will actually see them in the background to be inside your Keep notes but also you will see a confirmation like the following picture.

    LP Gmail to Keep Importer - Confirmation view

  11. You have successfully imported your Gmail messages into Google Keep :)

If you encountered any problems or your experience was different from what I described please contact me in order to fix the issue. As you will read later, the extension is highly experimental and it might not be working as expected in all operating systems (I had some issues with Mac OSX but I tried to resolve them).

In addition, it goes without saying that the importer works for plain emails. If you try to import emails that have attachments or fancy HTML websites embedded only the visible text of the message will be imported. As of version v0.13 the first line of the email becomes the title of the note in Keep.

Deep dive into the juicy details

This section contains more details about the problem itself, its issues, and the approach that I used to solve it. If you discovered a better solution contact me or even better do a pull-request at Github with your solution and I will adjust it accordingly.

Problem overview

Initially the problem at hand was to import Apple Notes into Google Keep. Apple Notes stores your notes as messages at Gmail (assuming you use a Google account of course) under the label Notes (yes, very innovative name by Apple that will not conflict with anything :D ). This makes it easier for us since the problem now can be reduced to importing from Gmail to Google Keep, which seemed trivial at first glance. Therefore, our problem now is two-fold: a) Fetch messages from Gmail and b) Create a note in Google Keep with code programmatically.

Gmail has a very nicely explained and easy to use Javascript API so the first task of fetching the emails under a label name is trivial and I will not go into more details since there are hundreds of articles explaining how to do this. Personally, I find Google’s documentation to be superb.

The problem is with the second part, creating notes into Google Keep. Google does not provide an open API for Keep as it does with Gmail so far, maybe because it is a new (couple of years) service or just because they don’t want developers yet to create apps around it. As a web developer myself I instantly had an idea to overcome this problem. Let’s use the Keep website and use the existing form programmatically, thus typing the note, the title and clicking the Done button, all with code. Nifty idea right? :)

The journey inside Google Keep website

My first goal was to come up with a way to identify and reference the three major components we need:

  • Title text box
  • Content text box
  • Done button (more problematic since the actual text might be different in each language)

Identification - Selectability

I dived into the HTML source code of the website, always using the awesome Chrome web developer tools, and I tried to detect something distinctive for each one of them. After trying several things the only solution that worked for me since I did not want to use actual text due to translations, is using CSS classnames. The only concern I had, have, and will have regarding this is that the classnames are not actual words so I guess Google uses some kind of builder to generate them, so at any moment in the future when Google modifies the Keep website there is a high probability that the classnames will change, thus breaking the plugin. Since I could not find any other identifiable characteristic I decided to go with it and in case this happens in the future I will just update the extension with the new names, until Google release a proper API.

Add note with code

So far so good, we have a way to reference each HTML element on the website. Let’s try to add some dummy content into the text boxes and simulate the click on the submit button to see what happens. 

1. Adding text programmatically: Works
2. Clicking Done: Does not work!

Hmmm, this was not expected at first because the button was clicked, but nothing happened and the note disappeared instead of being added. After diving deeper into the code I was examing the HTML dom of the elements in question throughout a proper manual note creation and saw what changed, where the content was added in reality, and what was the button’s behavior. I discovered that Google uses two elements per input field instead of one in order to allow expandable text areas. Interesting I thought, so I did a little research and found this great article by A List Apart which explains in more details how this is done.

Attempt number two. Having seen the issue I know referenced the proper element for the input boxes when adding the text, the ones which are really used during the submission of the note. Let’s try again.

1. Adding text programmatically: Works
2. Clicking Done: Does NOT work!

OK! Google Keep vs Lambros, 2 - 0.

Something was going wrong right? I went over my code several times to make sure I was not being stupid, and strangely enough I wasn’t so the problem was somewhere else. I spent several hours investigating the problem and I had an idea. Maybe it was something special happening when the submit button was pressed in reality that I was missing by doing it with code. Again, Chrome developer tools to the rescue.

I discovered a great tool that allows you to inspect the fired HTML/Javascript events of the page and even put breakpoints in your code to the exact point when it happens. If you want to see this and test it:

  • Open the developer tools in Chrome
  • Click the Sources tab
  • You can see on the right-hand side now a section named Event Listener Breakpoints
  • Play with it :)

Fast-forward a few hours and I discovered that in order to have a successful note submitted the button when clicked does not only fire the click event, but it has to also fire the mousedown followed by the mouseup events, in this specific order. Additionally, the same has to happen with the input boxes. All the events I use as of now to make a proper submission are shown below.

  • Input boxes (title, content)
    1. Add the text to the right element
    2. Fire events: change => mousedown => mouseup
  • Submit button
    1. Fire events: mousedown => mouseup

The implementation source code of this logic and the whole note creation is in the chrome-logic-keep.js file at Github.

Attempt number three. 

1. Adding text programmatically: Works
2. Clicking Done: Works

And voila! We can now add notes into Keep programmatically.

After this, I spent several hours figuring out how Chrome extensions work and went ahead and created my first extension. If you encounter any errors, please contact me or contribute to the project on Github. I have already said that it is a highly experimental project and it might break at any time if even the smallest detail changes on Google Keep’s website.

Google Keep vs Lambros, win for Lambros with KO!

As Jeff Bezos says, “It’s Still Day-One”

It was a very nice little project, and learned tons of new stuff. Web is amazing and ever-changing so we have to always keep learning.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Migrate to AWS - Make a static website using S3, Cloudfront and Route 53]]> https://www.lambrospetrou.com/articles/migrate-to-aws-static-website/ migrate-to-aws-static-website Thu, 11 Feb 2016 00:00:00 GMT A few months ago I started studying most of the Amazon Web Services (AWS) and as I said in a previous post one of my goals for this year is to transition all my apps and microservices into AWS.

This tutorial is the first part of a series and explains in detail how to setup a static website with unlimited throughput, durability and availability using Amazon S3. Additionally, we exploit Amazon Cloudfront to provide caching and super-fast downloads all over the world and finally, we use Route 53 to serve the website behind our custom domain name (e.g. lambrospetrou.com).

The reason I decided to write this post is because although the documentation in AWS is very good, the majority of all the articles and threads in several forums are a bit outdated and do not cover the new features of the AWS services we are going to use. For example, Cloudfront’s ability to serve GZIP-ed compressed content and its free HTTPS support. The reason is that Amazon releases hundreds of features every year and it is impossible for everyone to catch-up ;)

I will try to structure this tutorial in a way that each part is self-contained and builds upon the previous part, in order to allow you to pick-and-choose only the parts that you want for your website.

Overview

The most important service when it comes to static content is Amazon S3, which stands for Simple Storage Service. This is one of the best web services available as of this moment and not just among Amazon’s offerings but other competitors’ too. It is a file storage solution that offers super high-availability, extreme durability (eleven nines, 99.999999999%) and it is very cheap. Amazon S3 alone allows you to have a proper static website ready in a few minutes.

The other two services, Cloudfront and Route 53 are going to extend our website to provide caching in edge locations all over the world, to allow users to download the files from locations closer to them, and with Route 53 we will allow the usage of custom domains instead of using the default AWS ones (e.g. lambrospetrou.com.s3-website-region.amazonaws.com).

Make a static website using Amazon S3

Before we dive into the step-by-step guide let’s create our scenario.

Assume that throughout this tutorial we want to make a website for lambrospetrou.com. Also we want the www-prefixed domain, www.lambrospetrou.com to redirect the users to the APEX domain, the non-www domain, lambrospetrou.com.

  1. First we want to create the bucket that will hold our website files. Visit the AWS management console and navigate to the S3 service. (direct link to S3).

  2. Click Create Bucket and type in the Bucket Name, which in my case it was lambrospetrou.com. It is very important that you give your bucket a name that matches exactly your domain. Additionally, choose the region where you want your bucket to reside (choose the one closer to your users).

  3. Click Create and your new bucket should be visible under the list of All Buckets.

  4. Now click on the bucket you just created and select Properties from the tabs in the right side of the dashboard (you should see options like the picture below).

    Amazon S3 - Bucket properties

  5. The next step is to make this bucket act like a website.

    • Click on the Static Website Hosting option and select Enable Website hosting.
    • Type index.html in the Index Document option (and optionally fill the Error Document option if you plan to have an error page.). This option specifies that when a user navigates to a directory we want the index.html file of that website to be downloaded.
    • Click Save.

  6. Now the last step is to add permissions to the bucket to allow anyone to access its files.

    • Click the Permissions option.
    • Click Add bucket policy and paste in the following snippet (remember to replace lambrospetrou.com with your bucket’s name):
    {
    "Version": "2012-10-17",
        "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::lambrospetrou.com/*"
        }
    ]
}
    • Click Add CORS Configuration and paste in the following snippet:
    <?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="https://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>
    • Click Save. The above snippets allow everyone to request all files inside this bucket.

  7. That’s it! You have now a working website without the need to provision any server or any other infrastructure to support your static website. Whatever you upload in this bucket is going to be accessible from all over the world and rest assured that your website will always be online ;)

  8. The link to use in order to access your website is under the Static Website Hosting option, where it says Endpoint. It should be something similar to lambrospetrou.com.s3-website-region.amazonaws.com. I will refer to this endpoint throughout the rest of the tutorial as S3-Endpoint, so take a note of it.

  9. Try to open that link using your browser. Of course you will see nothing since you haven’t uploaded anything to your bucket yet ;p

Make the www-subdomain to redirect to the APEX domain

In this section I will describe how we can make all requests to www.lambrospetrou.com to be redirected to lambrospetrou.com. Again we only need to use Amazon S3 for this part. The idea is very simple. We will create another bucket named www.lambrospetrou.com but now instead of putting files into this bucket we will specify redirection rules to the bucket we created before.

  1. Create another bucket following the previous steps 1-4 but this time use the www-prefixed domain (e.g. www.lambrospetrou.com) as the bucket name.

  2. The next step is to add the redirection rules.

    • Click the Static Website Hosting option.
    • Select Redirect all requests to another host name
    • Fill the input box named Redirect all requests to: with your non-www domain name, which is also the bucket name your created before (e.g. lambrospetrou.com like the picture below).

    Amazon S3 - Redirect requests

    • Click Save

  3. Finished!

Test your website

  1. Upload your website to the non-www bucket you created first, or just create an index.html file with the following snippet:

    <html>
    <head>
        <title>Awesome website on AWS S3</title>
    </head>
    <body>
        <h1>Hello world!</h1>
    </body>
</html>

  2. Try to access your website using your S3-Endpoint (lambrospetrou.com.s3-website-region.amazonaws.com). You should see your website, if not there is a problem so contact me if you cannot figure out what step went wrong.

  3. Try to access your www-prefixed endpoint and make sure that you are redirected to the non-www one.

Use a custom domain with our website instead of the Amazon S3 endpoints

In this section, I will describe how you can use your domain name (e.g. lambrospetrou.com) to point to your website on S3 using Route 53, which is Amazon’s DNS solution offering.

  1. Register your domain with any registrar or Route 53 itself if you do not have a domain in hand.

  2. Open the Route 53 management console (direct link).

  3. Click Hosted zones from the left navigation menu.

  4. Click Create Hosted Zone in order to setup our domain.

    • For Domain Name you have to specify the non-www domain (e.g. lambrospetrou.com)
    • Click Create

  5. Open the hosted zone you just created and make sure that it has the NS and SOA record sets.

  6. Now you have to set the nameservers of your domain to the servers specified under the NS record. This is required only if your registrar is not Route 53. You can continue with the tutorial but in order for you to be able to see the changes you have to change the nameservers of your domain.

  7. Now we have to creare a Record Set for our domain in order to point to the S3 bucket we created above.

    • Click Create Record Set
    • Leave the Name as blank (empty).
    • Set the type to A - IPv4 address
    • Select Alias - YES
    • Type into the Alias Target your S3-Endpoint for your non-www bucket (e.g lambrospetrou.com.s3-website-region.amazonaws.com). VERY IMPORTANT: do not select from the suggested targets, type the full S3-Endpoint yourself (I will explain later on why). If this gives you an error about an invalid value, i.e Alias target contains an invalid value then use the S3 website region endpoint for alias target, like s3-website-eu-west-1.amazonaws.com. (note the dot at the end!).
    • Click Create.

  8. Repeat step 7 in order to create a Record Set for your www-prefixed domain but now you have to use www as the Name of the Record Set, and the www-prefixed S3-Endpoint as an Alias Target.

  9. You should allow a few minutes (or hours) for your changes to propagate through the network and assuming that your successfully changed the nameservers of your domain you should be able to visit your naked domain, lambrospetrou.com, and see your website as you would through the S3-Endpoint. Also, when you visit the www-prefixed domain you should be redirected to the naked one.

You can stop now if you want since you managed to create a static website that is very durable, highly-available, cheap, and that uses your custom domain. The rest of the tutorial will cover Amazon Cloudfront and how caching can affect your users all over the world.

Use Amazon Cloudfront to provide super-fast latencies all over the world

Again, we will use Cloudfront to provide a cache layer in front of our website for several reasons.

First of all, Cloudfront is a CDN (content delivery network) with edge locations scattered around the world which means that our users will always download the files from locations close to them avoiding long routing trips to the other side of the world (the location of your website is the region you specified during the S3 bucket creation).

In addition, Cloudfront is very useful because it allows us to use GZIP compression on most of the static files out-of-the-box, which is a new feature released in 2015 (related link).

Another very important advantage is that we can use HTTPS to access our website. By default S3 supports only HTTP requests through the custom domain, although using the S3-Endpoint allows for HTTPS. With Cloudfront we can use HTTPS from the user to AWS and then Cloudfront internally will use HTTP to communicate with S3 and return the content back to the user over HTTPS again. This feature in conjuction with the fact that now Cloudfront can be associated with an Amazon SSL key allows us to use HTTPS for Free. Both features were released in January 2016 (AWS Certificate Manager, Cloudfront Origin Security Features).

Let’s go ahead and create our first Cloudfront distribution.

  1. Open the Cloudfront management console (direct link).

  2. Click Create Distribution and then click Get Started under the Web section (see picture below).

    Amazon Cloudfront - Delivery Method

  3. Now you have to be very careful here.

    • Option Origin Domain Name should be set to the full S3-Endpoint of your non-www bucket (e.g. lambrospetrou.com.s3-website-region.amazonaws.com)
    • Leave the other options default for now since you can change these later as your website evolves (see two pictures below).

    Amazon Cloudfront - Origin Settings

    Amazon Cloudfront - Cache Settings

    • The only option you might want to change for now is the Forward Query String depending on whether your website uses query strings from users. By default Cloudfront will just strip out the query string before forwarding the request to S3 so enable this if your want it.
    • Additionally you will most probably want to enable the option Compress Objects Automatically since it will GZIP most of the static files, thus leading to even faster download times.

  4. In the Distribution Settings choose the Price Class you want depending on your user-base but bear in mind that the more edge locations you support the higher prices you pay.

  5. In the Alternate Domain Names (CNAMEs) you have to write in the custom domain names you want to use to point to this distribution. In my case I just have the naked domain lambrospetrou.com.

  6. The next step is to setup our HTTPS certificate (if you do not want HTTPS just skip this step and leave the defaults)

    • Select Custom SSL Certificate under the SSL Certificate option.
    • If you have already uploaded your own custom certificate to AWS use that, otherwise click Request an ACM certificate to get an Amazon certificate for free.
    • In the opened page type in your naked and www-prefixed domains (e.g. both lambrospetrou.com and www.lambrospetrou.com, see picture below).

    Amazon Certificate Manager - CNAMEs

    • Click Review and Request and make sure to confirm the certificate activation.
    • Now you should be able to select the newly created certificate from the dropdown menu (sometimes you might have to refresh the page to make the certificate available to the dropdown)

  7. Click Create Distribution and wait for the status to become Deployed instead of In Progress.

  8. Once the distribution status is Deployed use the Domain Name of your distribution to access your website (e.g. xxxxxxxxxxxx.cloudfront.net). If we did everything right you should now be able to see the exact page as if you visited your website through your domain or through the S3-Endpoint.

Setup a Cloudfront distribution for the www-prefixed domain

Now we need to create another distribution for the www-prefixed domain in order to use the second bucket we created that just redirects to the first.

  1. Repeat steps 1-6 but now use the www-prefixed domain in steps 3 and 5.

  2. Once your distribution is deployed try to visit it using the distribution’s Domain Name and make sure that the redirection works properly.

Set your domain in Route 53 to point to Cloudfront distribution

The last step is to make our custom domains (e.g. lambrospetrou.com and www.lambrospetrou.com) point to the Cloudfront distributions rather than the S3 buckets directly.

Since we have already setup Record Sets in Route 53 we just need to update them.

  1. Open the Route 53 management console (direct link).

  2. Click Hosted Zones from the left navigation menu.

  3. Select your domain hosted zone (e.g. lambrospetrou.com)

  4. Click on the naked (non-www) domain record set and update the Alias Target now to have the Domain Name of the corresponding Cloudfront distribution (e.g. xxxxxxxxxxx.cloudfront.net). Click Save Record Set.

  5. Repeat step 4 for the www-prefixed domain to use the second Cloudfront distribution.

Finally, Done!

Congratulations, you have created a website that is impossible to hack, it will always be available, it will always have exceptional performance, it is amazingly easy to update, you have HTTPS, you have GZIP compression for static files, you use your custom domain names and it should not cost you more than a few dollars (mainly for Cloudfront) unless you have billions of visitors, in which case I guess you make enough money with the content of the website ;)

The most interesting part of this architecture is that although this scenario uses entirely static websites, you can very easily extend this to support dynamic websites too and at the same time keep the caching layer for all the benefits it brings. You will just need to create another Cloudfront distribution that will handle a specific prefix of your domain, or even a subdomain, and which will point to your server (EC2, Elastic Load Balancer, etc.) in order to create the dynamic content of the website. This scenario and much more are explained in high detail in an amazing whitepaper by AWS which you can download for free named Hosting Static Websites on AWS. It covers the topics we implemented in this tutorial along with more advanced topics like A/B testing, user sharding, logging, security and many more.

I hope you liked the tutorial, but even if you did not I would really appreciate your feedback to improve it (I am pretty sure you can find where to send me a message).

Yes, my website at the moment (as of the time of this article) is hosted on S3, with Cloudfront distributions as defined above and Route 53 to handle my custom domain.

Notes

  1. The reason that we use the full S3-Endpoint when we refer to it either from Cloudfront distributions or from Route 53 Record Sets is simple. When S3 acts like a website, like in our scenario, it handles automatically the redirects and the retrieval of index.html files when the request is done on a directory. In order to use these features when behind of a custom domain we have to use the full S3-Endpoint otherwise if we use the links suggested in the AWS documentation or those suggested form the dropdown menus in the console these features will not work.

  2. You can play along with the caching behavior of Cloudfront distributions. With the default values, if you do not specify a Cache-Control header on an object in S3, the defaults are going to be used which based on the documentation use 1 day of caching. Depending on your website you can specify to cache for less time, forward query strings, forward custom headers or not, etc.

  3. You can use the S3-Endpoint during development for easier testing since Cloudfront will serve the cached version to you which will annoy you during testing ;p Otherwise you can invalidate some files in your distribution, but even that takes time so think carefully about your caching technique (times, versionings, etc.) and use S3-Endpoints to ease up your dev process.

References

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Create a contact form using Amazon Web Services (AWS) Simple Notification Service (SNS)]]> https://www.lambrospetrou.com/articles/create-contact-form-using-aws-sns/ create-contact-form-using-aws-sns Wed, 28 Oct 2015 00:00:00 GMT A common feature among most websites is a way to contact the owner for any comments or feedback (or complaints). In previous versions of my website and blog I used several techniques to provide a contact me form. In almost all the cases I had to implement the actual email submission on the website’s hosting server or in some cases used a 3rd party service that provided me this functionality by querying an HTTP endpoint they gave me with the actual message and they sent the email for me, to me.

Today I decided to roll out my own contact-form service using AWS. As it turns out, it is amazingly easy to setup and it only takes you 15 minutes from the moment you sign up for the free-tier account to the point where you have an active form sending you emails. Awesome time we live in!

Assumptions

  • You have an active AWS account (if not do yourself a favor and register for the 1-year free tier at AWS).
  • You have a website somewhere or you know how to use Simple Storage Service (S3) from AWS to host your website files.

Services

In order to implement the contact form we are gong to use two services from AWS.

  1. Simple Notification Service (SNS) (SNS service) Amazon SNS is a fast, flexible, fully managed pub-sub messaging service. Use it as a cloud-based mobile app notification service to send push notifications, email, and SMS messages; or as an enterprise-messaging infrastructure. (description as per AWS documentation)

  2. Amazon Cognito (Cognito service) Amazon Cognito gives you unique identifiers for your end users and then lets you securely store and sync user app data in the AWS Cloud across multiple devices and OS platforms. You can do this with just a few lines of code, and your app can work the same, regardless of whether a user’s devices are online or offline. When new data is available in the sync store, a user’s devices can be alerted by a silent push notification so that your app can sync the new data automatically. (description as per AWS documentation)

Procedure

The whole procedure is very simple. We will create an SNS topic (imagine it as an endpoint) where the website will send the message of the contact form. As soon as SNS receives the message it will automatically send it over to our email, which we will have configured through the AWS console to be a subscriber of that specific topic. Amazon Cognito is used just as a way to get temporary credentials for the website in order to be able to publish the message to the SNS topic.

I will explain the implementation in three steps, a) Amazon Cognito, b) SNS and c) Securing our SNS topic to allow only publishings from our website domain.

Amazon Cognito

We use Cognito just for the sake of getting temporary credentials for our website in order to be able to use any AWS service through the Javascript SDK.

  1. Login to the AWS console and open the Amazon Cognito service (direct link)

  2. Select Create new identity pool

    Amazon Cognito - Create new identity pool

  3. Type in a name for your pool (i.e. Website_Contact_Form) and enable Unauthenticated identities.

  4. You should have something along the lines of the following picture.

    Amazon Cognito - Identity pool information

  5. Click Create Pool and then on the next page you get the message “Your Cognito identities require access to your resources”. Just click Allow (bottom right) to proceed.

  6. You should be in the Edit identity pool page now where you see the information you provided above. From the left side menu choose Sample code and in the Platform dropdown select Javascript.

  7. Download the AWS SDK since you will use it on your website later.

  8. You can use the code provided to get temporary credentials for the newly created identity pool. In my case the code is similar to what is shown below.

    // Initialize the Amazon Cognito credentials provider
AWS.config.region = 'eu-west-1'; // Region
AWS.config.credentials = new AWS.CognitoIdentityCredentials({
    IdentityPoolId: 'eu-west-1:e4c24108-5050-42f8-ac0b-761c46aa947f',
});

That’s it for the Cognito service!

Simple Notification Service (SNS)

SNS is a bit longer procedure but nothing complex. The philosophy behind SNS is that you have a topic (endpoint) with some publishers and some subscribers. When a publisher posts something to that topic then all the subscribers are notified automatically by the service with the newly published message. I guess by now you should have figured out that the subscriber to our SNS topic will be our email and the publisher of the topic will be our website.

Let’s dive in!

  1. Login to the AWS console and open the SNS service (direct link)

    SNS - Create Topic

  2. Select Create Topic and then just fill out the Topic name with whatever you want (i.e. com-website-contact-form). You should now see the Topic Details page for your new topic (similar to the picture below).

    SNS - Topic details

  3. Click on the dropdown Other topic actions and select Edit topic policy.

  4. From the Edit topic policy screen make sure you are in the Basic View and modify the options as per the picture below and click Update. Basically we allow everyone to become publisher of the topic but only we (the owner) can subscribe to it.

    SNS - Edit topic policy

  5. Now you successfully have a topic that anyone can publish messages to.

  6. Time to add our email as a subscriber to the topic. From the Topic Details page, click on the Create Subscription button.

  7. Make sure to change the Protocol to Email and fill out your email on the corresponding box and click Create Subscription. An email will be sent to your email in order for you to confirm that you would like to subscribe to this topic.

    SNS - Create Topic Subscription

Perfect, we are almost there!

Code

Here I will provide the very minimal code required to publish a message from your website. I will not include any styling or validation or anything since the scope of the article is just to connect the different AWS services together.

Below you can find the whole HTML and Javascript code used to send a message to your email from the website.

<html>
<head>
    <script src="aws-sdk/dist/aws-sdk.min.js"></script>
    <script type="text/javascript">
        var LPAWS = {};

        // Initialize the Amazon Cognito credentials provider
        AWS.config.region = 'eu-west-1'; // Region
        AWS.config.credentials = new AWS.CognitoIdentityCredentials({
            IdentityPoolId: 'eu-west-1:e4c24108-5050-42f8-ac0b-761c46aa947f',
        });

        LPAWS.sendToTopic = function() {
            var sns = new AWS.SNS();
            var params = {
                //Message: 'Hello topic', /* required */
                Message: document.querySelector('#input-msg').value,
                Subject: 'Browser SNS publish - contact form',
                TopicArn: 'arn:aws:sns:eu-west-1:717437904155:com-website-contact-form'
            };
            sns.publish(params, function(err, data) {
                if (err) console.log(err, err.stack); // an error occurred
                else     console.log(data);           // successful response
            });
        };
    </script>
</head>
<body>
    <form>
        <input type="text" name="msg" id="input-msg"/>
        <button onclick="LPAWS.sendToTopic(); return false;">Send to SNS topic</button>
    </form>
</body>
</html>

I deployed locally the above code and as you can see from the screenshot below it works perfectly! Good for us!

End-to-End test working

Secure our SNS topic

The only issue we have right now is that anyone can publish messages to the SNS topic. Therefore we could receive a lot of spamming if someone inspected our Javascript code and extracted the endpoints for the SNS topic and Cognito identity pool.

The solution to this problem is to add a policy rule in our SNS topic to only allow our website to act as a publisher, which is trivial.

  1. Login to the SNS console (direct link)

  2. From the left side menu select Topics

  3. From the list of topics select the one we created before, then click at Actions and from the dropdown select Edit topic policy.

    SNS - Secure policy

  4. Now instead of the Basic View you should go to the Advanced View

  5. In Advanced View you can directly edit the topic policy rules in detail. We want to restrict access for publishing only to our domain. Make sure that the second statement of the rules is as the following excerpt.

    {
  "Sid": "__console_pub_0",
  "Effect": "Allow",
  "Principal": {
    "AWS": "*"
  },
  "Action": "SNS:Publish",
  "Resource": "arn:aws:sns:eu-west-1:717437904155:com-website-contact-form",
  "Condition": {
    "StringLike": {
      "aws:Referer": [
        "https://www.lambrospetrou.com/*",
        "https://lambrospetrou.com/*"
      ]
    }
  }
}

    The above configuration adds the Condition to the existing rule which is what is restricting access to the Publish permissions of the topic only to our domain. Make sure that you use your own Resource arn url which can be found in the topics details page and also your own domains.

  6. Done. Your website is the only one with rights to publish messages into your SNS topic (except of course yourself through the console). Please be advised though that it is not difficult for programmers to create TCP packets with the proper headers to fool this policy. At least we make it a bit harder for the usual user :)

Conclusion

In conclusion, we have seen how easy it is with Simple Notification Service (SNS) and Amazon Cognito services to create our own mailing system. You can extend this configuration to send SMS, Push notifications to your devices or anything else you can imagine just by playing with the different SNS subscription types.

There are many features built into SNS that you can use in order to make your messages more advanced and more powerful. You can read all about them in the AWS SNS documentation.

If you spot any errors in the tutorial or your experience is different from what is described here please don’t hesitate to contact me.

Disclaimer

The details used above for the SNS topic and Cognito identity pool were deleted after completing this tutorial.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[After-school effect]]> https://www.lambrospetrou.com/articles/after-school-effect/ after-school-effect Sun, 25 Oct 2015 00:00:00 GMT Time passes and goes at amazing speed. Finally, after twenty-five years, school has reached its end. I still don’t know if this is bad or good, since I am now thrown into the wild jungle known as life.

Anyway, enough with the philosophical talk. Now that I graduated from university I have some time to revisit some of my old projects and refactor them. As always, when I want to try something new I re-write my website. During last week I completely re-wrote my website and blog from scratch, lambrospetrou.com.

As I described in my last article, dated in 2014-July-20, I am in love with Go aka #golang so it was a natural choice to use it again. I decided to go away from the dynamic nature of a blog and revert to a static blog with dynamic content ;) Well, let’s just say that whenever I create write an article I regenerate the website. Static Site Generators have seen a lot of popularity increase in the last couple of months, maybe because they are easy to use or because the site is easier to deploy than messing around with Wordpress, Joomla, plugins and all the sh**t brought with them. Additionally, as a Software Engineer I am always fond of my own creations. You can find the generator for this blog at Github: Micro-blog generator repository. I have to warn you that it is not exactly designed for generic usage. However, if you like its simplicity and want to try it just check out the source code of this blog at Lambros Petrou blog source code and replace it with your own content.

Apart from my website and blog, another important project I had in pending state was Spi.To, my own custom URL shortener and text share tool. During last year, I was experimenting with Polymer and Paper elements to give it a material design makeover. However, I found it to be pretty unstable (it was still in development stages so it might have fixed its issues with its current stable version) and very heavy for the simple and minimalistic product I wanted. I ported most of the stuff I needed into native Dart and now I find it to be much more responsive.

I plan to add more features to Spi.to during the coming months and I really want to replace the back-end with a server-less infrastructure at Amazon Web Services (AWS). AWS is among my top priorities in the To-Learn list for this year and I would love to create more projects utilizing the myriad services available by Amazon, especially the new API Gateway, Lambda functions and of course the awesome Simple Storage Service (S3).

One thing I wanted to do for a long time was to find a way to edit my CV (resume) easily without booting into Windows just to use Microsoft Word and without searching for a Latex template that would most probably not match the design I wanted. I decided to turn to the easiest platform for customizations and easy updates, the web. I replicated the design of my old CV using only HTML and CSS, leading to a pixel-perfect document which is amazingly easy to update and to customize in-detail. You can find my new CV version at lambrospetrou.com/cv/.

In conclusion, I would like to take advantage of my time working at Amazon and be certified as an AWS Certified Professional, which will boost my knowledge and skills around cloud computing and at the same time open many more doors career-wise. As it seems AWS is the dominant cloud computing provider and it has no plans to cease existing for the foreseeable future.

It is nice to be in production mode again :)

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Micro-blog service written in Go as a personal blog]]> https://www.lambrospetrou.com/articles/micro-blog-service-written-in-go-personal-blog/ micro-blog-service-written-in-go-personal-blog Sun, 20 Jul 2014 00:00:00 GMT I have been a user of Wordpress since 2006 in order to provide my own blog, latest version found at MasterGenius.NET. I invested a lot of time in it learning PHP, web development (HTML, CSS, Javascript and the likes) and I thank it since it made my way into the web as a developer. There have been many times I wanted to switch out from it and make my own micro-blog service but due to time constraints and other things popping up I always halted the process.

Recently, I started using Go language as my primary tool for my latest projects and I am really amazed by how much you can create in a small amount of time and at the same time having a substantial performance. All other languages that tried to do something similar failed either in performance or in coding-style, at least for my taste. Go is the sweet spot ;)

I wanted a project in order to learn Go further than reading and doing some tutorials so the micro-blog service project kicked-off. This blog is the result of those 3-4 days and I am going to use it as a more personal blog with thoughts and opinions on variety of topics and also post some useful tutorials (without going too much out-of-topic though like my previous blog).

This blog is written entirely in pure Go #golang language without any 3rd party library (i.e gorilla/mux or martini) that many Go articles suggested and I have to admit that I am very impressed. The core standard library is very sufficient and provides you all the tools to create anything you want easily. As a back-end I am using Couchbase server 2.x which is one of the best NoSQL databases I have tried so far, and speedy-demon too. As for the content of the articles I made a very minimalistic and simple panel where I can write them in Markdown format. I have chosen Markdown for the articles because it lets me write in plain text without any HTML tags or CSS classes getting involved, thus I can concentrate on the content. And it is good to know since it is being used in many other services too, like Github. The Markdown content is compiled in HTML when posted by the Go back-end.

The old blog is still live, at least for now, and I will try to find some noteworthy tutorials I wrote back in the days and post them here too, like I already did with some of them but I will keep this blog cleaner and more personal.

I hope you like this new blog and if you have any suggestions don’t hesitate to contact me ;)

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Wordpress permissions guide]]> https://www.lambrospetrou.com/articles/edit-wordpress-permissions/ edit-wordpress-permissions Mon, 18 Nov 2013 00:00:00 GMT If you have ever installed wordpress on your own server, at some time you faced permissions problems. Especially when you are migrating a wordpress blog from one host to another and there is file copying involvement through ftp, scp etc.

If not, lucky you! :)

Scenario

Description Value
username lpuser
apache2 groupname www-data
wordpress dir /home/lpuser/wordpress/

Let’s say you are lpuser and your server that runs the wordpress blog belongs to the www-data permissions group.

Wordpress Permissions

Everything should have the 0755 permission for the user, apart from the .htaccess files and the wp-admin/index.php file that need to have 0644.

Let’s do this:

sudo chmod -R 0755 /home/lpuser/wordpress/
sudo chmod 0644 /home/lpuser/wordpress/.htaccess
sudo chmod 0644 /home/lpuser/wordpress/wp-admin/index.php

Now if you execute the following command you should verify that everything has rwxr-xr-x, and the two other files rw-r--r--.

If you try now to make changes to your theme through the wordpress editor then you won’t be able to save since the server does not have write permissions. We can find that apache runs under the user www-data and the same group, as seen by the following commands:

ps aux | grep apache
groups www-data

So an easy solution to fix the problems is to add ourselves to the group www-data and then make the files to be owned by us, and the new group we belong.

This can be achieved with the following commands:

sudo usermod -a -G www-data lpuser
sudo chown -R lpuser:www-data /home/lpuser/wordpress/wp-content/themes
chmod -R 0775 /home/lpuser/wordpress/wp-content/themes

The last command allows the group to read/write/execute on the themes folder only.

If you have another way don’t hesitate to share it. The scenario is that we have a username different than the server’s that runs the wordpress blog.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Share your Dropbox folder between Windows and Ubuntu on dual-boot system]]> https://www.lambrospetrou.com/articles/share-dropbox-folder-windows-ubuntu-dual-boot/ share-dropbox-folder-windows-ubuntu-dual-boot Thu, 16 Aug 2012 00:00:00 GMT Many users are dual booting Windows and Ubuntu for numerous reasons today. Well a problem that may arise is that if you have Dropbox, you should have 2 folders of your Dropbox files, 1 for Windows and 1 for Ubuntu.

Following the guide below you can consolidate them into one and reclaim your wasted space back.

NOTE: I am assuming that the Windows partition is mounted at startup when you boot into Ubuntu.

Guidelines

  1. Set up your Dropbox folder inside Windows first and verify that it is syncing your files without problems.

  2. Boot into Ubuntu and stop Dropbox from syncing.

    pkill dropbox

  3. Delete your Dropbox folder from Ubuntu (the command below assumes it’s located in your home directory).

    rm -rf /home/_username_/Dropbox/

  4. Create a symlink to your Windows Dropbox folder (this will create a symbolic link in your home directory that links to your Dropbox folder in Windows).

    ln -s /media/WindowsOS/User/_username_/Dropbox /home/_username_/Dropbox

    /media/WindowsOS/User/_username_/Dropbox the path to the Dropbox folder in your Windows partition mounted in Ubuntu

    /home/_username_/Dropbox the default folder for Dropbox to synchronize

  5. Reboot and you’ re finished. Now you maintain only one copy of your Dropbox folder which needs to be synced only once, no matter which OS is booted at any time.

That’s it, I hope you won’t find any problems. If you do though, contact me :)

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Install JDK 7 (Java Development Toolkit) on Ubuntu 12.04 LTS]]> https://www.lambrospetrou.com/articles/install-jdk7-ubuntu-12_04/ install-jdk7-ubuntu-12_04 Fri, 27 Jul 2012 00:00:00 GMT Since version 7 of Java Ubuntu no longer has in its packages the required files for the Oracle JDK 7 due to new license terms by Oracle. The default option for Ubuntu is OpenJDK but if you really want the standard Java from Oracle (former Sun) then you have no easy way to do it on your own.

Therefore I prepared a simple script that will install the Java 7 development toolkit on your system with no effort at all.

  1. Download the correct package from Oracle website: JavaSE Downloads.
    Choose the JDK version and afterwards Linux x86 or Linux x64 depending on your system. It must be the file with the .tar.gz extension.

  2. Download my script from here: Oracle-JDK-Installer Repository
    Click the button ZIP and after downloading it extract it anywhere, and remember the path.

  3. Put the package you downloaded at step 1 in the same directory as the jdk-installer.sh you downloaded (step 2).

  4. Type

    ```bash
sudo ./jdk-installer.sh PATH_TO_PACKAGE SYSTEM_ARCHITECTURE
```

    You can read the instructions at my github repository (step 2) for more information.

  5. Verify your installation:

    java -version

    You should get a message mentioning the version you downloaded.

  6. Visit Verify Java Version and click Verify Java version in order to check that the browser plugin works.

That’s it folks!

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Control the fan speed of you Thinkpad T430 running Ubuntu 12.04 LTS]]> https://www.lambrospetrou.com/articles/control-thinkpad-t430-fan-ubuntu-12_04/ control-thinkpad-t430-fan-ubuntu-12_04 Fri, 20 Jul 2012 00:00:00 GMT After researching for a few hours online to find a guide on how to control my Thinkpad’s fan speed I realized that the new models have some differences from previous models and the guides available are not complete if not wrong. So, I am making this tutorial for anyone that has a new Thinkpad (x30/x20 models) and needs to control his fan in order to keep the noise down and get more battery life.

Every step below uses the terminal so open an instance with the combination CRTL + ALT + T

  1. The first thing we will do is to install a program that will provide us information about the sensors of the laptop and their temperatures

    sudo apt-get install lm-sensors

    Configure the application in order to find every available sensor

    sudo sensors-detect

    Answer Yes to every question and the last confirmation for saving the changes made.

  2. Install thinkfan which is our main program:

    sudo apt-get install thinkfan

  3. Add the coretemp module to the startup list. It will provide us the temperature inputs.

    echo coretemp >> /etc/modules

  4. Load the coretemp module.

    sudo modprobe coretemp

  5. The next step is to find your temperature inputs so take note the results of the following command.

    sudo find /sys/devices -type f -name "temp*_input"

    If you don’t get any outputs ( similar to the next step ) please Reboot and continue from this step.

  6. It’s time to edit our thinkfan configuration

    sudo vim /etc/thinkfan.conf

    Go to the line where it says #sensor /proc/acpi/ibm/thermal … and below that line (which should be commented since thermal is not supported in the new thinkpads) insert something like the following:

    sensor /sys/devices/platform/coretemp.0/temp1_input
sensor /sys/devices/platform/coretemp.0/temp2_input
sensor /sys/devices/platform/coretemp.0/temp3_input
sensor /sys/devices/virtual/hwmon/hwmon0/temp1_input

    The above lines are the results from Step 5 prefixed with ‘sensor ‘.

  7. Time to set the temperature rules. The format is: FAN_LEVEL, LOW_TEMP, HIGH_TEMP meaning that each FAN_LEVEL will start when the highest temperature reported by all the sensors meets its LOW_TEMP and if it surpasses its HIGH_TEMP it will go to the next FAN_LEVEL rule. If it goes below the LOW_TEMP it will fallback to the previous FAN_LEVEL rule.

    Please take notice that the HIGH_TEMP of a rule must be between the LOW_TEMP & HIGH_TEMP of the rule that follows.

    My settings are:

    #(FAN_LEVEL, LOW, HIGH)
(0,    0,    60)
(1,    57,    63)
(2,    60,    66)
(3,    64,    68)
(4,    66,    72)
(5,    70,    74)
(7,    72,    32767)

    NOTE: I am not responsible for any problems you encounter with these rules. They are fine as per my configuration so please test them before using them and if necessary adjust them to your needs.

  8. Now, we must add a configuration file into the modprobe.d

    sudo echo "options thinkpad_acpi fan_control=1" >> /etc/modprobe.d/thinkpad.conf

  9. If you want to start thinkfan automatically at boot-time please type the following:

    sudo vim /etc/default/thinkfan

    Change the line START=no to START=yes. If the line does not exist, add it yourself.

  10. RESTART your laptop and everything should work as expected. Test your laptop’s temperatures (using sensors command) under different workloads and verify that the fan speed is as per the rules you defined.

If you encounter a typing mistake or a step not working for you please contact me.

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[nVidia Optimus Support in Ubuntu 12.04 LTS - How to get the most of your battery life guide]]> https://www.lambrospetrou.com/articles/nvidia-optimus-ubuntu-12_04/ nvidia-optimus-ubuntu-12_04 Fri, 20 Jul 2012 00:00:00 GMT Anyone that purchased a laptop with nVidia Optimus is facing problems with Linux support. In a few words, Optimus is not supported officially ( the best project so far is Bumblebee ). As a result the power draw is more than double compared to integrated card-only systems because both cards are enabled and loaded but only one is in use, the Integrated one.

Bumblebee is a project that tries to bring Optimus support in Ubuntu and so far it had good results but the battery impact was inevitable.

The latest update though changed that and now linux fans can have their Optimus configuration without the enormous battery draw. Follow the steps below to install bumblebee but please note that I am assuming you removed any nVidia drivers you may installed on your own.

  • We must add the repositories for the bumblebee packages so please type the following in your terminal CRTL + ALT + T

    sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
sudo add-apt-repository ppa:bumblebee/stable
sudo apt-get update

  • Now install the bumblebee:

    sudo apt-get install bumblebee bumblebee-nvidia

    NOTE: If you do not want the discrete card to be enabled at all please use the following command instead:

    sudo apt-get install --no-install-recommends bumblebee

    This will not install the drivers for your discrete card but if you want in the future to have it just enter the following and get it installed:

    sudo apt-get install bumblebee-nvidia

  • Some people recommend to reboot now although I saw the results without doing so.

  • If you installed the nVidia driver too, in order to launch a program and use the discrete card you must use the following command:

    optirun _appnamehere_ &

In order to get support for 32-bit applications with nVidia you should do the following:

sudo apt-get install virtualgl-libs-ia32

I hope that you will find this guide helpful. I got amazing results with it myself:
Power draw before: 26-28w
Power draw after: 11-13w

Bumblebee Project Website

]]> lambros@lambrospetrou.com (Lambros Petrou) <![CDATA[Change your Computer Name (hostname) inside your network - Ubuntu]]> https://www.lambrospetrou.com/articles/change-computer-hostname-ubuntu/ change-computer-hostname-ubuntu Sun, 11 Sep 2011 00:00:00 GMT Your hostname or otherwise Computer Name is the name that your computer uses inside the network it is connected to in order to distinguish itself. Usually you setup your own during the setup of the Operating System but if for any reason you want to change it check out the guide below.

  • Open a terminal CRTL + ALT + T

  • Execute:

    sudo vim /etc/hostname

  • Change it to whatever you want your name to be, Save and Close the file

  • Now we have to update our hosts file in order to route ourselves.

    sudo vim /etc/hosts

  • Find the line where it says: 127.0.1.1 usernameHere and change it to what you entered above and then Save and Close the file.

Enjoy your new Computer Name :)

]]> lambros@lambrospetrou.com (Lambros Petrou)